As per Gartner, “Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.” So basically, we are implementing our integration layer, which we used to have on-premises, on the cloud.
The scenario I’m going to implement here, although basic, will demonstrate how several cloud products integrate to do the job. The integration layer will accept entities (such as a customer, vendor, etc…) and saves the entity into a database. Although this seems trivial at first sight, on the cloud special considerations should be taken. The integration layer needs to be resilient, performant, extensible, robust, and most importantly reliable. On the cloud, certain assumptions that can be safely assumed on-premises, is not the case on the cloud. Any part of the puzzle can be down, or not accessible in a certain point of time. Therefore, some cloud design patterns need to be followed here, such as asynchronous communication, retries, and queuing.
To demonstrate the idea in a clear manner, I’ve depicted a high level architecture diagram of the solution I’m going to follow:
So here, our iPaaS layer is everything between the Web App and the SQL database. The Web App (or any other client), posts a message to the Service Bus exposed via the API Management as a normal Web API. This is an asynchronous call, a fire-and-forget one. After the message is persisted in the Service Bus topic, we define an ‘n’ number of subscriptions to cater for different types of messages, for example, a subscription to handle customers, another one to handle vendors, etc. A Logic App will be listening on any of the subscriptions to handle the business process. The Logic App will call an Azure Storage Table to get some configuration values, will call an Azure Function to Generate the record Id, then will call a Web API application that is responsible to call the Azure SQL database to persist the record.
If the insertion failed, the Logic App will send a message to a service bus queue, where another Azure Function is listening. This function will pick up the message and send the failure information to the initiator via an email.
If the insertion succeeds, the Logic App will post the result back to the Web application via a service exposed in API management.
As you see, such an implementation, although can be enhanced in many aspects, achieves the below:
- Performance: Client calls the iPaaS asynchronously, meaning immediate response and no waiting threads.
- Resiliency: If any component is down, the message is not lost and will be picked up and acted upon once the component is up again
- Extensibility: Additional message types and handling logic can be added to handle any new data entity types
- Robustness: The iPaaS layer is able to cope with errors since when a message is read from the service bus, it’s not removed, it’s only peeked and locked, so in case of any errors, the message will be safely unlocked and will be available to be picked up later by subscriptions
- Reliability: The system will always behave according to specifications. We can safely ensure that no messages will be lost and once a message is posted to the service bus, there can be either one of two outcomes, either we send a success response back to the client when the record is inserted, or we send a failure email.
Please note that I will not explain each of the used products in details. This is a practical-oriented series that will touch on the needed features of each product. For detailed explanation on each Azure product, you can refer to the Azure official documentation on https://docs.microsoft.com/en-us/azure/