Architecting enterprise integrations
Integration problems faced by each organization are different. For some organizations, it may be to implement a few point-to-point connections like creating data entries in SAP ERP system based on the sales events in Salesforce. In other cases, it may be to interconnect inventory, sales, and distribution systems to offer a mobile app for online purchasing. In addition, these integrations may be developed by a central team or multiple independent teams. Some integrations may have very high scalability requirements while others may be cost-sensitive.
Architectures that are best suited for integration problems also differ based on the nature of integrations. In this article, we will explore some architecture patterns that can be used for different integration projects.
Components of integration architecture
Before going into details about integration architectures, it’s useful to understand commonly used components in integration architectures and their applications.
Integration platforms:
This is the main component of integration architectures. Integration platforms execute integration flows and provide a collection of connectors and protocols to integrate with other systems. They also include advanced data transformation capabilities to map data structures among different systems.
API Management:
Exposes integrated services as APIs by providing authentication, access control, rate limiting, and usage monitoring for service invocations.
When to use: If integrations are exposed to external parties or other business units.
Identity and Access Management (IAM) systems:
Provides advanced security features for the integration deployment.
When to use: API Managers also provide some level of security features such as authentication and access control. IAM systems can be used if it is necessary to support more advanced security features such as multi-factor authentication, access control policies, and fully-fledged user management.
Registry:
Provides a centralized location for storing integration artifacts.
When to use: Certain integration artifacts such as endpoints and data schemas may be reused across multiple integrations. In addition, endpoints can change due relocation of services. In such cases, it is helpful to store these integration artifacts in a registry and configure all integrations to look up those from the registry.
Messaging platforms:
Provides publish-subscribe and multicast delivery of messages with message persistence.
When to use: If the recipients of a message need to be changed frequently and if the message losses while processing need to be avoided.
Observability components:
Provides monitoring and analytics capabilities to the deployment such as usage dashboards, message flow analysis, log analysis, resource consumption metrics, and notifications based on pre-configured thresholds.
When to use: Useful for understanding the status of the deployment and to take preventive actions.
With this knowledge about components of integration architectures, now we can explore different architectures and their usages.
Centralized integrations
In this architecture, a central integration platform is used for deploying integrations. API Manager and other components of the architecture are also deployed as central deployments. The central integration platform as well as other components are usually deployed as clusters of two or more instances to facilitate scalability and high availability. A centralized integration architecture is shown below:
Two API gateways are used in this architecture to handle external and internal traffic. This enables enforcing network security policies such as allowing external requests only for the external API gateway. All components in the integration architecture publish observability data such as usage metrics, error details, and logs to the observability component so that it can provide a comprehensive view of the system. In practice, this observability component may consist of multiple systems such as ELK for log analysis, Prometheus for runtime metrics, and Jaeger for tracing.
When to use centralized integrations
Centralized integrations are useful when all integrations of an organization are governed by a central team. It is still possible for multiple teams to develop integrations. However, the deployment of integrations is either performed or approved by a central team. Operational aspects of integrations are also handled by this central team. This architecture facilitates better and simpler governance of integrations and usually has lesser infrastructure requirements. A main disadvantage of this approach is the reduced agility of integration development as integration developers have to depend on a central team for the deployment.
Microservices-style integrations (Micro integrations)
In this architecture, integrations are deployed as microservices. A small number of integrations related to a single integration task, usually developed by a single team, are deployed into an integration runtime. Multiple integration runtimes are deployed depending on the number of integration tasks and development teams. API management components can also be deployed according to the microservices style (i.e. micro API gateways). Usually, these integration runtimes are specially designed to consume lesser resources and to support deployment on containerized environments such as Kubernetes. The below diagram depicts a microservice-style integration architecture.
Three integrations are shown in the above architecture, and each integration is deployed in its own integration runtime. Integrations 1 and 2 are exposed to the outside via micro API gateways. As each API is deployed in its own API gateway, these can also be managed independently by the teams that develop corresponding micro integrations. Integration 3 only connects backend systems, so it is exposed to the outside.
When to use micro-integrations
Micro integrations are useful when it is necessary to facilitate more independent integration developments among a large number of teams. As it reduces bottlenecks introduced by centralized governance processes, integrations can be taken to production quickly and improved incrementally. Some of the main drawbacks are the requirement of container platforms such as Kubernetes for managing integration runtimes and the need for implementing complex monitoring solutions for obtaining overall analytics about integrations.
Mixing centralized and micro-integrations
This architecture facilitates both centralized and micro integrations. Therefore, the integration layer of this architecture consists of a central integration platform and multiple micro-integration runtimes. API layer can also have combinations of centralized API gateways and API micro gateways. This architecture is depicted in the figure below:
As shown in the diagram, micro-integrations can act as composable services to central integrations. Furthermore, micro-integrations can be exposed to central integrations via micro API gateways or messaging systems, to facilitate authentication, access control, and reliability.
When to use the combined integration architecture
Two categories of integrations may exist in large organizations: 1) integrations that connect lower-level services, which are usually accessed by a single business unit, and 2) integrations that span multiple business units and access sensitive services. In this scenario, the first category of integrations can be deployed as micro integrations, which can be developed and taken to production within the business unit. The second category requires more governance with approvals from multiple stakeholders and access to multiple services. Therefore, these integrations can be implemented in a centralized manner.
Closing remarks
The architecture patterns discussed in this article can act as a guide for selecting the proper architecture for integration projects. However, the selection of the architecture and components within the architecture needs to be done by considering factors such as the nature of integrations, non-functional requirements, organization size, available infrastructure, and budget.
