Integration is about interconnecting different systems to work together to produce desired outcomes. Programming is to write a set of instructions to perform some tasks. Traditionally, these two were considered to be different concepts. But, what if we can combine these two concepts…

Photo by Helloquence on Unsplash

When integrating systems, we need to connect remote servers over different protocols such as HTTP, JMS, and FTP. Various security mechanisms have to be supported and enforced among connected systems, such as TLS, OAuth and SAML. Systems may communicate using different message formats, which have to be transformed to enable interoperability. We have to take care of occasional failures of servers and reliable message passing within such failure prone systems. Furthermore, we have to deal with asynchronous communications, where a response for a request may arrive in a separate channel at a later time. In order to address above and other numerous problems in enterprise integrations, a new stack of software was developed, namely the middleware stack. This stack consists of many components, either bundled together as a single monolithic product or shipped separately as different products such as Enterprise Service Buses (ESB), message brokers, workflow systems, etc. Each of these components usually has its own way of configuring and interacting with external entities. For example, ESBs can be programmed by using a product specific configurations (mostly XML based) to state which endpoints to call, how to transform a message, etc. Message brokers may provide JMS (API level) or AMQP (wire level) as a method of interacting. A workflow engine may support BPMN as a method of specifying steps to follow in a business process. With such stack, integration is about deploying these components and configuring them to perform a required integration scenario. Then the problems mentioned above will be taken care by those components with little effort from the integration developer. For example, almost all ESBs have a built in method to transform messages just by providing an straightforward configuration (e.g. XSLT file). Similarly, most workflow engines provide a way to wait for asynchronous messages and correlate them with a relevant flow.

Now let’s consider traditional programming.. mainly, we get a set of primitive constructs to define the program flow (if, for, while, etc) and work with variables (assignments, comparisons). In addition, we get some libraries to perform tasks such as threading, networking, database access, etc. Such libraries are again some programs written in the same language or a lower level language.

Now to our topic: can we program integrations? First, developing configurations for middleware products can be thought of as a kind of programming. In fact, many configuration languages used in such products offer many programming constructs such as conditional branching, loops, assignments in some form. Therefore, developing configurations involves defining the flow using such constructs and using another set of constructs for service invocations, controlling transactions, transforming messages, etc where necessary. This sounds very similar to programming! However, anyone would soon realize that it is not the case when applied in real use cases.

Most integration languages mainly focus on providing constructs for common integration tasks, such as invoking services, message transformations, splitting/combining messages, etc. However, their support for flow control and variable manipulations is limited or involves too much code. This could be partly due to the design goal of such systems, whose objective is to provide a tool where users can drag and drop integration tasks and wire them to create a flow. Resulting languages (or configuration), mostly based on XML, produce complex and lengthy source code even for relatively simple integrations. Even though doable in some cases, writing complex variable manipulations or processing steps in such languages soon becomes unmanageable. Integration vendors have identified this issue and addressed this by adding support to embed code written in actual programming languages such as Java or Javascript within integration flows. With this approach, integration developers have to decide when to use what. One user may try to implement complex processing steps using the integration language, which could result in unreadable code. A more common observation is users trying to implement too much integration logic in embedded program snippets by writing tasks such as database access, transformations or even service invocations within custom code. Such behaviors hinder the benefits of using an integration platform and could result in erroneous, inefficient or insecure integrations.

Then to the next possibility.. can we use programming languages such as Java or C# to program integrations? Writing complex processing steps in programming languages is easier. Further, almost all programming languages have libraries for networking (TCP, HTTP, JMS, etc), database access, XML/Json manipulations and for many security related tasks. Therefore.. yes, it is possible program integrations in most programming languages. However, it is not that simple.. To begin with, most integrations have to listen for incoming requests. Which means you have to start a server within your code and take care of thread allocations. Then if you need to authenticate requests you may need to use an OAuth or SAML library and configure relevant data stores. When calling external endpoints, you have to consider their security policies such as TSL, OAuth, etc., and code in appropriate logic to build compliant messages. In addition, you have to handle invocation timeouts, failures and load balancing as needed. If your use case requires receiving responses in a separate channel you have to deal with message correlation and persistence. If you want to make sure that the integration flow does not cause inconsistencies due to failures in certain steps, you may have to somehow enforce a transactional behavior. Furthermore, there may be cases where you cannot afford to loose the progress of the work done in an integration flow, even if the runtime is crashed. Therefore, although possible, it is not trivial to program integrations using a programming language.

Then what should be our solution for programming integrations? We have to come up with a new programming language where all integration tasks are first class language features. Such language should support all features of a generic programming language such as control flow constructs (e.g. if, for, while), variable manipulations (e.g. assignments, comparisons), multi threading, concurrency controls and ability to use external libraries. In addition, and most importantly, it should provide built-in support for integration features including:

  • Service invocations over many common protocols
  • Handling timeouts and failures in service invocations
  • Listening for incoming requests using multiple methods
  • Applying relevant security policies for incoming and outgoing messages
  • Message delivery guarantees
  • Asynchronous communication over multiple channels
  • Distributed transactions
  • Failure recovery of the language runtime
  • Analyzing the runtime and message flows

Of course, building such a language requires an enormous effort. In fact, many points listed above require a separate discussion on how such functions can be embedded into a language syntax and how to implement them in the language runtime. However, if we could build such a language with reasonably simple syntax, its benefits are also enormous…

Such language will allow any developer to build integrations. As integration functions are native constructs of the language, developers can just use them anywhere necessary with minimal effort. It also minimizes the types of systems needed. No longer, there is a need to deploy separate servers for each integration function. We just have to run programs written in our integration language. Now, our integration platform and the programming language being the same, we can develop any complex logic within our integration flows. We don’t have to switch to the hybrid approach of embedding code snippets within our integration flows.

With the introduction of such integration language, we get a wide range of deployment options. We can run the language runtime as a server and deploy integration programs in it. This is closer to what we have with current integration platforms from the deployment perspective, but with all the power of a programming language. What’s more interesting are the new deployment options enabled by this concept of “integration as a program”. We can deploy each integration flow as a separate runtime. i.e. run each integration flow/program separately, perhaps on different VMs. With this approach, integration deployments will be reduced to executions of few programs. Another approach would be to run a new program for each request. Although state maintenance can be a problem in this approach, when backed with a scalable data store, this could be the best approach for deployments with very high performance requirements.

Therefore, if we have a programming language for integrations, it could change how we think about integrations and open up whole new ways to address problems of application integration. Now, the natural question would be whether there are such integration languages? In order to explore this area, we have been working on an integration language for few years and introduced the Ballerina language as the result of this effort.

PhD, Software Architect, Academic, Works @ WSO2

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store