Architecture for digital transformation

Organizations undertake digital transformation initiatives to utilize technology as much as possible to support business operations. Therefore, many applications have to be developed for various business operations carried out by an organization such as inventory management, procurement operations, supplier management, payroll handing, advertising / sales campaigns, building space management and vehicle fleet management.

State of a typical organization

Even before a digital transformation project, organizations use many IT systems. These can be commercial systems, which may be deployed on-premise data centers, on cloud VMs (e.g. AWS, Azure, etc) or consumed as SaaS systems (e.g. Salesforce, NetSuite, etc). Further, an organization may have some custom developed systems for certain specific business operations (e.g. developed using Spring Boot, .NET, Go, etc). In addition, an organization may have multiple databases, containing information about different entities and applications.

Although most of these systems can connect with a central user store (e.g. LDAP/AD) to get user information, each application has to authenticate users before offering its services. Each of these systems will be providing some services (e.g. as APIs) and those service interfaces have to be protected from unauthorized and uncontrolled access. Furthermore, as most of these systems do not exist in isolation and need to interact with other systems, many integrations need to be performed within the deployment. Finally, when a deployment gets larger and more complex, monitoring and trouble shooting tasks become increasingly complicated.

Outcome of digital transformation

As the outcome of a digital transformation initiatives, it is necessary to integrate these different existing systems, data stores, business processes and physical entities (where possible) and offer a seamless user experience tailored for organization’s business operations. If we take an example of a supermarket chain, below are few possible use cases expected from such initiative:

• When inventory levels run low, purchasing orders have to be automatically sent to registered suppliers (steps such as selecting suppliers and approving large purchase orders may need human intervention).
• Once a delivery is ready, logistics providers should be assigned automatically (again some human approval steps might be needed) to transport goods from suppliers to warehouses or to deliver online orders made by customers.
• When a new employee is recruited, her details should be propagated to POS systems, payroll system, procurement systems, etc based on assigned roles.
• Web portals and mobile apps should be available for customers, store staff, management staff and suppliers to perform relevant operations.
• When a customer is signed into one business application, he should be able to use services of all business applications without signing in again.
• All business service usages by customers, staff and partners should be monitored and controlled.

Most importantly, it should be possible to introduce new services to the organization’s IT platform and integrate those with existing services with a minor effort.

Architecture

Now let’s consider a generic architecture for building an IT platform for such digital transformation initiatives (Figure 1).

Figure 1: Architecture for digital transformation platform

Let’s consider main components of this platform:

Integration layer

This layer performs all integration activities among on-premise backend systems, file-based systems, data stores and SaaS systems. It can perform protocol bridging, message transformations, message enrichment, content validation, service orchestration, etc in order to implement integration flows among backend systems.

This layer is deployed within the internal network and secure connections (e.g. VPN, protected endpoints, etc) should be available for it to integrate systems that are not deployed within the data center. Usually components of the integration layer can be scaled as necessary to match the load (e.g. auto-scale by utilizing the capabilities provided by infrastructure such as Kubernetes pod auto-scaling and AWS auto scaling groups). Furthermore, it is also possible to deploy more than one integration clusters depending on the requirements for segregating business services and load.

API gateway clusters

API Gateways intercept incoming traffic to the deployment in order to enforce security and other policies as well as to capture API usage statistics. It is possible to deploy one or more API gateway clusters as necessary in order to isolate API traffic based on different client parties (e.g. general users and partners) and enforce relevant network security policies (e.g. limit partner API gateway to IP ranges of partners).

API Gateways are usually deployed within the internal network and incoming traffic is routed to API gateways via a load balancer placed within the DMZ. However, it is also possible to deploy external facing API gateways in DMZ depending on the organization’s policies. Similar to the integration layer, API gateway clusters can also be auto-scaled by utilizing auto-scaling capabilities provided by the infrastructure.

API management layer

API management plane facilitates API publishing, policy definition, application registration, API subscription and API life cycle management activities. This layer interacts with API gateway clusters in order to publish API definitions and to exchange API authorization information. As the API management layer usually provides web portals for API creators, API users and administrators, it may be necessary to facilitate access via a load balancer placed within DMZ (e.g. if external users need to subscribe for APIs).

Identity and Access Management (IAM)

IAM layer provides user management, authentication and authorization (policy evaluation) functions to the whole deployment. Below are some of the specific features an IAM layer can provide related to above areas:

• Support for OpenID Connect and SAML2 for authentication and exchanging user information
• Support for authorizations based on OAuth2 / XACML
• Single Sign-On (SSO) to enable multiple services to be accessed without having to authenticate with each services (typically implemented using OpenID Connect or SAML2)
• Multi-Factor Authentication (MFA) to enhance the authentication process (e.g. password and SMS OTP based authentication).
• Conditional or context based authentication (e.g. users in store-admin role are allowed to authenticate only during office hours and if connecting using a certain IP address range).
• Support for connecting with multiple user stores such as LDAP/Active Directory and RDBMS
• Connecting / federation with external identity providers to authenticate users who are not registered in the organization’s IAM systems (e.g. sign-in using Google, Facebook, etc or external IAM systems).
• Web portals, mobile apps or other interfaces for users to perform self sign-up, profile management, password recovery, etc.

Usually, IAM layer is also deployed within the internal network and clustered as necessary to meet scalability and high-availability requirements. As IAM layer provides portals/interfaces for users, it may be required to provide external access to it via a load balancer placed within the DMZ.

Business Process Management (BPM)

Some business operations require many interactions with users at multiple steps. Examples would be selecting suppliers, approving large orders, handling customer complaints, etc. These operations may have one or more approval steps or steps that require user input. Such business operations (or parts of those operations) may have to be implemented as workflows modeled in standard workflow languages such as BPMN or BPEL.

BPM systems support deployment, execution, administration and analysis of such human oriented workflows. Once a BPM system is deployed, it can be integrated with other business systems and user portals via the integration layer. Furthermore, similar to any other business system, BPM systems can also utilize SSO and other IAM features provided by the IAM layer (e.g. for assigning users to workflow tasks).

Observability layer

All systems in the deployment, including the API layer, integration layer, IAM layer, BPM systems as well as backend systems, should provide mechanisms to monitor their behaviors. This includes collecting and analyzing logs, collecting and summarizing operational metrics and tracing message flows across multiple systems. Observability layer performs these tasks by collecting logs, metrics and traces from all systems and providing a unified view of the deployment. Therefore, if something goes wrong in the deployment, administrators don’t have to inspect each and every system individually, as they can obtain all information by just logging into the observability layer.

Providing business services across multiple locations

Organizations may have user bases distributed across multiple geographical locations (e.g. in multiple states or countries). In such cases it may be necessary deploy business services in multiple data centers or cloud regions closer to client locations in order to satisfy performance and regularity requirements. Architecture discussed here can be extended in such scenarios to support multiple data centers/cloud regions as shown in figure 2. In this case, usually only the API gateway clusters and integration layer clusters have to be deployed in additional data centers. API management plane, IAM layer, BPM systems and the observability layer deployed in the main data center can be used by all additional data centers.

Figure 2: Extending the deployment across multiple data centers

As API gateways and integration components handle the main runtime load of the deployment, having a single deployment of other components does not affect the performance in most cases. Furthermore, having a single management and monitoring layer naturally supports such tasks to be carried out centrally.

Utilizing the architecture for developing business applications

So far, we have discussed an architecture that can be use for digital transformation projects. Now we can look at how this architecture can be used to build business applications.

When developing a business application, we can first think about the APIs that need to be exposed. This can be modeled in Open API specification and deployed in the API layer. It is possible to deploy APIs as prototypes so that client application developers can start testing them and provide feedback.

Then we can focus on the back end implementation. Usually, many new applications can be composed using existing business services. In such cases, we can implement most parts of the new application in the integration layer. If the new application requires complex logic, in addition to integrations, we may have to implement some parts in a programming platform (e.g. Spring Boot, Go, .NET. etc) and use integration layer to integrate it with rest of the systems. Another option is to use an integration focused programming language such as Ballerina, so that complex logic and integrations can be implemented in the same platform.

All client applications and backend systems can use the IAM layer to perform authentications (e.g. SSO for client apps) and authorizations, without implementing those features within each application. Similarly, all applications can publish logs, metrics and traces to the observability layer to enable centralized monitoring of the deployment. Finally, it is necessary to maintain a CI/CD pipeline to build, test and deploy all artifacts related to new applications, so that API definitions, application logic and integration artifacts can be maintained in a source control system and deployed via a controlled deployment flow.