EDA Implementation — Exposing Event Streams as Managed APIs

Photo by Mark König on Unsplash

Note: This is part 3 of a three-part article series on Event-Driven Architecture implementation methods. Following are the previous articles on this series: Part 1, Part 2

Organizations can generate a large number of events during business operations such as reception of sales orders, complaints filed by customers, notifications of inventory levels, temperature readings from warehouses, etc. Multiple processing steps need to be executed in order make these events useful and integrate those with existing business systems as we have discussed in previous articles [1][2].

Once we process and extract useful information out of event streams, we need to make those available to relevant users and other systems. Such consumers of processed event streams may be within the organization or can be outside organizational and network boundaries (e.g. partners, customers, mobile apps, etc). Therefore, it is necessary to enforce security and other policies, and properly monitor and govern event consumption. Asynchronous APIs and supporting API management platforms are useful in such scenarios to manage events as APIs. This is depicted below:

In following sections, we will look at different ways of exposing events as Async APIs by using WSO2 API Manager as an example.

WebSocket APIs

WebSocket protocol provides following advantages over protocols such as AMQP, Kafka, MQTT, etc, which makes it suitable for exposing event streams to external parties.

• WebSocket is compatible with HTTP(S) and can work over most networks that support web traffic (e.g. Internet)
• Most of the popular web browsers support WebSocket protocol, which enables web pages to interact with event streams to perform tasks such as updating graphs based on real-time data, implement chat sections in pages, etc.
• As WebSocket allows two systems to communicate multiple two-way messages over a single TCP connection, connection initiator can also publish or receive events. Therefore, systems that don’t have public IPs can also publish and receive events (e.g. mobile apps and web pages).

When exposing WebSocket streams to internal or external consumers, we may need to enforce certain policies such as:

• Who can discover and subscribe to a WebSocket endpoint
• Who has to approve a subscription to a WebSocket stream
• Who can publish and subscribe to a given WebSocket topic
• How many events can a single application/user published to or received from a given WebSocket topic
• The duration which an application can consume events from a given WebSocket stream

In addition, it is necessary to monitor consumers of different event streams and event flows among publishers and consumers. Let’s consider the below scenario and see how we can use Async API support in WSO2 API Manager to implement it:

HLine is logistics company which operates multiple warehouses across the country. Multiple suppliers and retail companies use HLine’s warehouses to distribute and order goods. HLine’s IT systems record orders placed by retail companies and maintains current inventory levels in each warehouse. This information is published continuously to a WebSocket stream.

HLine’s management needs to get real-time information about these statistics via web portals and mobile apps. However, only the procurement managers should access the inventory level information and sales managers should access the order details.

In addition, there are many temperature and humidity sensors installed in each warehouse, which publish reading to a central system. Such monitoring event publications should be allowed only for authorized devices and rate of publishing needs to be restricted to 50 events per minute to avoid overloading the central system.

Let’s assume that HLine’s inventory management system has an internal WebSocket server running at http://internal-hline.com:8090. This server has three topics for event streams that we discussed above:

• ws://internal-hline.com:8090/inventory — Publishes inventory levels of items
• ws://internal-hline.com:8090/orders — Publishes order details
• ws://internal-hline.com:8090/monitoring — Receives monitoring data from devices

Usage of WSO2 API Manager in this scenario is illustrated in the below diagram:

Inventory management system’s websocket server is fronted by the API gateway. Mobile applications and monitoring systems connect to the API gateway (which is accessible over the Internet) to consume and publish events.

We can create a WebSocket API in WSO2 API Manager to expose these topics as below:

Now we can specify required security policies for each topic. Below screenshot shows the association of an scope (i.e. permission) named “inventory_data” to the “/inventory” topic. “inventory_data” scope is allowed only for the “procurement_manager” role.

In order to restrict monitoring event rate to 50 events per minute, we can associate a rate limiting policy with the Monitoring System as below. This policy will restrict each client system to publish only 50 events per minute.

After creating the API, it is possible to view the AsyncAPI definition, which is automatically generated by the platform:

WebHook APIs

WebHook is another protocol that can be used for stream based communications. This protocol has three entities: Hub, Publishers and Subscribers. Hub can maintain any number of topics. Publishers can send HTTP messages to any topic. Subscribers can subscribe to any topic by specifying the topic name and a HTTP callback URL. Whenever a message is sent to a topic, it will be broadcasted to all subscribers. Hub performs this broadcasting by calling the callback URL of each subscribers with the message payload.

It has below characteristics:

• As WebHook message flows are HTTP based, it can seamlessly work with existing HTTP servers and clients.
• WebHook protocol does not maintain an open TCP connection between publishers (or the hub) and consumers, which makes it more efficient for situations where the event rate is very low (e.g. once a day)
• WebHook event consumer should have a callback URL, which needs to be accessible from the Hub. Therefore, consumers need to have public IPs if it is necessary to receive events over Internet (i.e. not suitable for mobile apps).

Similar to WebSockets, WebHook event streams can also be managed as APIs by enforcing authorizations on each WebHook topic, controlling the number of subscribers, restricting the consumable event rate for each application/user, governing the API creation and subscription process, etc. Below diagram shows a WebHook based architecture with API Management:

Here, API Gateway acts as the Hub to which event consumers can subscribe. It is possible to create any number of topics per WebHook API during the API design phase. Event publishers can publish to any authorized topic using the HTTP URL of the topic. Below screenshot shows designing of a WebHook API topic in WSO2 API Manager. Note that the topic authorization is restricted to “product_details” scope. Runtime section shows event publishing URLs (HTTP and HTTPS) for this topic.

Server-Sent Events APIs

Server-Sent Events (SSE) can be used to send one-way event streams over HTTP connection from an event publisher to a consumer. Once the consumer initiates the HTTP connection, event publisher (i.e. server) keeps the connection alive and sends multiple responses (i.e. events). Below are some characteristics of SSE:

• As SSE is based on HTTP, it can be used with most of the existing web infrastructure (e.g. can be used to send event streams over the Internet)
• Most web browsers support SSE, which makes it suitable for displaying real-time updates on web pages, etc.
• As event consumers initiate the connection, SSE can be used by devices that do not have public IPs (e.g. mobile apps)
• SSE can only handle one-way event stream, so that connection initiator cannot publish events (e.g. IoT devices operating without public IPs cannot use SSE to publish events)
• SSE maintains an open TCP connection between the publisher and the consumer. Therefore, it more suitable for relatively short periods of interactions with higher event rates.

As with other streaming protocols, SSE event streams can also be governed by API management. This facilitates enforcement of topic level access control, subscription count and event rate restrictions, controlled API publications and monitoring of SSE based event streams. Generic architecture for SSE API management with WSO2 API Manager is shown below:

All API specifications and policies are defined in the API management plane (i.e. all-in-one component). API gateway acts as a proxy between SSE event sources (i.e. backend systems) and SSE consumers. This enables API gateway to enforce all defined policies on SSE streams and collect usage data to be summarized and visualized in the analytics component.

For example, below screenshot shows a policy specification section of for an SSE API in the API portal. Event consumers subscribing for this API can select one of the three enabled policies: unlimited events, 5000 events per month with 5000 subscriptions and 1000 events per month with 1000 subscriptions.

As we have discussed in this article series, EDA projects need to consider multiple aspects such as fetching events, integrating event sources and consumers, transforming event contents, aggregating multiple event sources, filtering and summarizing event data and governing event stream interactions. In most cases these requirements need to be facilitated by specialized tools including event brokers, integration products, stream processing engines, event-enabled API platforms and by using appropriate protocols.