Applying Event-Driven Architecture in Digital Transformation Projects

We have to consider a wide range of use cases in digital transformation projects. Some of these use cases are short-term interactions such as a user or a system invoking a service and expecting an immediate response. An example would be to getting the list of products under a certain category in a shopping portal. However, there can be a significant number of use cases that require more complex interactions with multiple systems as we discuss below.

A customer placing an order in a shopping portal

Multiple internal and external systems as well as shopping company’s staff could be involved in this process. Each of involved entity must receive relevant order details in a reliable and secure manner. Some steps in the process may take a long time to complete (e.g. packing items) and some steps are hard to scale based on the load (e.g. shipping packages).

Monitoring buildings

An office building can be equipped with light sensors, motion sensors, thermometers, etc. Theses sensors will be continuously transmitting their readings, so that relevant systems can use those as necessary. E.g. fire control system can check abnormal increases in temperature or security system can check for suspicious movements within usually unoccupied time periods.

On-boarding an employee

HR staff fills in employee details in the HR portal. This activity should update multiple other systems such as payroll system, procurement system, leave management system, etc.

Now let’s consider some properties of above and similar use cases:

• Temporary failures or unavailability of involved entities should not result in failures of the use case (e.g. temporary unavailability of the inventory service should not fail the order process).
• Temporary higher loads on the system have to be tolerated (higher number of orders during holidays may delay some orders, but should not fail any order).
• A large number of messages (or events) have to be processed by a considerable number of systems (e.g. various sensors in the building monitoring system may send their reading every 10 seconds)
• The entity that is sending an event (e.g. sensor or HR portal) may not know or care about the entities that are receiving the event (e.g. employee registration events sent by the HR portal may be consumed by multiple existing systems as well as by some of the new systems in future)
• Results of carrying out an use case cannot be notified to interested parties immediately (e.g. customer cannot be notified about the shipment of goods as soon as the order is placed).

Although this is not an exhaustive list, if business use cases with above requirements are present in a digital transformation project, it could be an indication that Event-Driven Architecture (EDA) has to be utilized in certain areas.

Architecture

Now let’s consider the below architecture for incorporating EDA within a scope of a large project.

Figure 1: Generic architecture for event-driven systems

This is essentially the same architecture we discussed under the generic digital transformation architecture. Here we have highlighted the EDA related components and protocols, and introduced some IoT related event sources (sensors, networked devices, smart buildings, vehicles with networked sensors, etc). Note that there can be many other event sources other than IoT devices. Any system within the deployment can emit events (e.g. HR portal sending an event when an employee is assigned to a different branch).

Events backbone

The main component we focus here is the events backbone or the messaging layer, which supports two messaging mechanisms: publish/subscribe and reliable messaging. Publish/subscribe mechanism allows interested parties to subscribe for topics, so that all subscribers receive a copy of each message published to those topics. Reliable messaging ensures that messages are not discarded until it is delivered to subscribers. RabbitMQ and Kafka are some commonly used events backbones.

Stream processor

Another important component in EDA is the stream processing engine. Multiple systems and devices can emit events continuously. However, there are many scenarios where such raw events do not make sense. For example, temperature readings sent every 10 seconds are not useful unless there is no change in the temperature value. Similarly, although a PoS system may send an event on each purchase, management dashboard system might be only interest in the total number of sales per item completed within each hour.

Figure 2: Stream processing script for summarizing total number of sales per item conducted within each hour (developed in WSO2 Streaming Integrator)

Stream processor performs such filtering, aggregation, pattern matching, etc operations on raw events as necessary before consumption by relevant business systems. WSO2 Streaming Integrator and Kafka Streams are some possible options here. Example WSO2 Streaming Integrator script for summarizing PoS sales data is shown in figure 2.

Edge processing components

In some scenarios we need to filter and aggregate events closer to the event sources in order to minimize the load on central deployments as well as on the network. This is a common requirement in IoT use cases. In such situations, stream processors have to be deployed out side the main data center (e.g. in factory floors, trucks, etc), possibly in resource constrained devices.

Furthermore, if it is necessary to communicate with devices using IoT related protocols such as MQTT, it may also be necessary to deploy MQTT brokers and scaled down integration layers (e.g. to convert MQTT to REST) closer to the devices. Edge processing devices contain such scaled down stream processors, integration layers and security implementations deployed closer to IoT devices.

Integration layer

Above architecture consists of an integration layer connecting all systems in the deployment. This is useful in EDA to bridge protocols between the events backbone and rest of the systems. For example, if a business micro service sends events in GRPC and the events backbone can only receive events in AMQP, integration layer can perform the protocol conversion. However, if the events backbone supports communicating over multiple protocols or if certain systems can communicate using messaging protocols (e.g. AMQP), it is possible to bypass the integration layer and communicate directly with the events backbone.

Applying the architecture

Now let’s consider building few use cases on top of this architecture.

Use case 1: Order processing

First, let’s consider the scenario of placing an order in a shopping portal. Some prominent components involved in this use case are shown below.

Figure 3: Application of EDA for order processing use cases

Now let’s consider the message/event flow of this use case:

1. Customer logs into the shopping portal by authenticating (i.e. SSO) with the IAM layer.
2. Once the customer completes the order, shopping portal sends a REST request to the API layer with an API token.
3. API layer validates the token and sends the request to the integration layer.
4. Integration layer converts the REST message to an AMQP message and sends it to the events backbone (e.g. to the “orders” topic).
5. BPM system, which is subscribed to the “orders” topic will fetch the order from the events backbone (if it can interact directly over AMQP) and starts processing it. If the BPM system cannot communicate via AMQP, integration layer has to subscribe to the “orders” topic and deliver messages to the BPM system over the required protocol (e.g. REST).
6. Whenever the BPM system needs to interact with any business service (e.g. inventory system, shipping system, etc), it will sends a message either to a topic in the events backbone or to the relevant business system (via the integration layer if needed).
7. Once the order processing is completed, BPM system sends an event to the “completed_orders” topic.
8. Integration layer, which has subscribed to the “completed_orders” topic receives the event and sends it to the shopping portal using web sockets.

Use case 2: Temperature monitoring

Key architecture components involved in the temperature monitoring use case are shown below:

Figure 4: Application of EDA in temperature monitoring

Let’s consider a possible events flow among these components:

1. Thermometer sends temperature reading to the edge processor over MQTT every 15 seconds.
2. Stream processor deployed within the edge processor averages these reading over 10 minute intervals and sends the the average value to “average_temperature” topic in the events backbone every 10 minutes.
3. In addition, edge processor’s stream processor checks for temperature changes above 2 degrees within a 10 minute interval. If such change is detected, it sends an event to the “temperature_changes” topic.
4. Integration layer has subscribed to the “average_temperature” topic, and it sends each average temperature value to web based monitoring portal over web sockets.
5. Management/admin system has subscribed to the “temperature_changes” topic, so that whenever a temperature change is detected it evaluates some rules (e.g. comparing with historical values) and decides whether an action is required. If action is needed, it sends an event to the “temperature_alert” topic.
6. BPM system has subscribed to the “temperature_alert” topic. When an alert event is present, it starts a workflow and assigns tasks for relevant staff members.
7. Monitoring portal has also subscribed to the “temperature_alert” topic (via web sockets through the integration layer). When an alert is present, monitoring portal displays that as an emergency message.

As we have discussed, it is possible to implement many event-driven use cases with this architecture. However, we have to decide which use cases and which parts of certain use cases needs to be implemented with EDA. Properties of event-driven use cases we discussed in the beginning of this article outlines some guidelines for this decision. In many deployments, we may have to combine both EDA and synchronous models depending on the business needs and characteristics of participating entities.