Applying Event-Driven Architecture in Digital Transformation Projects

We have to consider a wide range of use cases in digital transformation projects. Some of these use cases are short-term interactions such as a user or a system invoking a service and expecting an immediate response. An example would be to getting the list of products under a certain category in a shopping portal. However, there can be a significant number of use cases that require more complex interactions with multiple systems as we discuss below.

A customer placing an order in a shopping portal

Monitoring buildings

On-boarding an employee

Now let’s consider some properties of above and similar use cases:

  • Temporary failures or unavailability of involved entities should not result in failures of the use case (e.g. temporary unavailability of the inventory service should not fail the order process).
  • Temporary higher loads on the system have to be tolerated (higher number of orders during holidays may delay some orders, but should not fail any order).
  • A large number of messages (or events) have to be processed by a considerable number of systems (e.g. various sensors in the building monitoring system may send their reading every 10 seconds)
  • The entity that is sending an event (e.g. sensor or HR portal) may not know or care about the entities that are receiving the event (e.g. employee registration events sent by the HR portal may be consumed by multiple existing systems as well as by some of the new systems in future)
  • Results of carrying out an use case cannot be notified to interested parties immediately (e.g. customer cannot be notified about the shipment of goods as soon as the order is placed).

Although this is not an exhaustive list, if business use cases with above requirements are present in a digital transformation project, it could be an indication that Event-Driven Architecture (EDA) has to be utilized in certain areas.

Architecture

Figure 1: Generic architecture for event-driven systems

This is essentially the same architecture we discussed under the generic digital transformation architecture. Here we have highlighted the EDA related components and protocols, and introduced some IoT related event sources (sensors, networked devices, smart buildings, vehicles with networked sensors, etc). Note that there can be many other event sources other than IoT devices. Any system within the deployment can emit events (e.g. HR portal sending an event when an employee is assigned to a different branch).

Events backbone

Stream processor

Figure 2: Stream processing script for summarizing total number of sales per item conducted within each hour (developed in WSO2 Streaming Integrator)

Stream processor performs such filtering, aggregation, pattern matching, etc operations on raw events as necessary before consumption by relevant business systems. WSO2 Streaming Integrator and Kafka Streams are some possible options here. Example WSO2 Streaming Integrator script for summarizing PoS sales data is shown in figure 2.

Edge processing components

Furthermore, if it is necessary to communicate with devices using IoT related protocols such as MQTT, it may also be necessary to deploy MQTT brokers and scaled down integration layers (e.g. to convert MQTT to REST) closer to the devices. Edge processing devices contain such scaled down stream processors, integration layers and security implementations deployed closer to IoT devices.

Integration layer

Applying the architecture

Use case 1: Order processing

Figure 3: Application of EDA for order processing use cases

Now let’s consider the message/event flow of this use case:

  1. Customer logs into the shopping portal by authenticating (i.e. SSO) with the IAM layer.
  2. Once the customer completes the order, shopping portal sends a REST request to the API layer with an API token.
  3. API layer validates the token and sends the request to the integration layer.
  4. Integration layer converts the REST message to an AMQP message and sends it to the events backbone (e.g. to the “orders” topic).
  5. BPM system, which is subscribed to the “orders” topic will fetch the order from the events backbone (if it can interact directly over AMQP) and starts processing it. If the BPM system cannot communicate via AMQP, integration layer has to subscribe to the “orders” topic and deliver messages to the BPM system over the required protocol (e.g. REST).
  6. Whenever the BPM system needs to interact with any business service (e.g. inventory system, shipping system, etc), it will sends a message either to a topic in the events backbone or to the relevant business system (via the integration layer if needed).
  7. Once the order processing is completed, BPM system sends an event to the “completed_orders” topic.
  8. Integration layer, which has subscribed to the “completed_orders” topic receives the event and sends it to the shopping portal using web sockets.

Use case 2: Temperature monitoring

Figure 4: Application of EDA in temperature monitoring

Let’s consider a possible events flow among these components:

  1. Thermometer sends temperature reading to the edge processor over MQTT every 15 seconds.
  2. Stream processor deployed within the edge processor averages these reading over 10 minute intervals and sends the the average value to “average_temperature” topic in the events backbone every 10 minutes.
  3. In addition, edge processor’s stream processor checks for temperature changes above 2 degrees within a 10 minute interval. If such change is detected, it sends an event to the “temperature_changes” topic.
  4. Integration layer has subscribed to the “average_temperature” topic, and it sends each average temperature value to web based monitoring portal over web sockets.
  5. Management/admin system has subscribed to the “temperature_changes” topic, so that whenever a temperature change is detected it evaluates some rules (e.g. comparing with historical values) and decides whether an action is required. If action is needed, it sends an event to the “temperature_alert” topic.
  6. BPM system has subscribed to the “temperature_alert” topic. When an alert event is present, it starts a workflow and assigns tasks for relevant staff members.
  7. Monitoring portal has also subscribed to the “temperature_alert” topic (via web sockets through the integration layer). When an alert is present, monitoring portal displays that as an emergency message.

As we have discussed, it is possible to implement many event-driven use cases with this architecture. However, we have to decide which use cases and which parts of certain use cases needs to be implemented with EDA. Properties of event-driven use cases we discussed in the beginning of this article outlines some guidelines for this decision. In many deployments, we may have to combine both EDA and synchronous models depending on the business needs and characteristics of participating entities.

PhD, Software Architect, Academic, Works @ WSO2

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store