In this digitalisation era, where most enterprises are going online, cloud-based integration platforms, namely Integration Platforms as a Service (iPaaS) are becoming a must and hence gaining a lot of traction. One iPaaS that offers its clients cloud-based services is SnapLogic. In this article, getting started with SnapLogic, we introduce this platform, explain its architecture, main components, and highlights.
SnapLogic, like other iPaaS solutions, makes it easy for companies to connect cloud-based applications and on-premise enterprise platforms. The product is designed to make it easy to ingest and integrate data from diverse sources using a no-code / low-code interface that is suitable for technical as well as citizen integrators.
Whatever your integration requirements, SnapLogic should be able to perform them. All forms of integration, including batch, real-time, and streaming integration, are supported by the SnapLogic architecture. This makes it easier to handle data of any type, speed, or volume.
SnapLogic Architecture
SnapLogic uses cutting-edge and adaptable architecture. A self-updating and elastic runtime called Snaplex serves as the foundation of the design. Snaplexes can be either called a Cloudplex or a Groundplex. Both are practically the same thing except that Cloudplexes are hosted on the cloud and offered by SnapLogic where they are completely managed and scalable. On the other hand, Groundplexes are self-managed Snaplexes that are hosted and managed by the client. If the client has requirements of integrating both cloud and on premise systems, a hybrid approach of Cloudplexes and Groundplexes can be used.
The SnapLogic “Intelligent Integration Platform” (IIP) is the control plane cloud-based web portal that is used to automate all phases of the integrations, including designing, developing, deploying, and maintaining them. Note the term “Intelligent” in the name of the platform. That is because SnapLogic employs an Artificial Intelligence (AI) technology called Iris that aids in the development of your integrations by suggesting the next steps that could be used based on millions of metadata elements and billions of data flows metadata that SnapLogic has through its large customer base. The nice thing is that this AI technology has been there for SnapLogic since years, hence, it’s very mature and is not created and promoted lately as part of the AI hype only.
As you can see in the above architecture diagram, SnapLogic IIP is the control plane, Cloudplexes and Groundplexes are the runtimes where integrations are hosted (data plane). Cloudplexes helps in integrating cloud-based applications, while Groundplexes are used to integrate enterprise applications deployed behind the company’s firewall. Data on the Snaplexes doesn’t make its way into the IIP, only metadata does, so data security is preserved.
Note: One more note, If you are familiar with Boomi (one of the most famous iPaaS platforms), there’s huge similarities between both of these iPaaS’s in terms of architecture, where effectively Snaplexes are synonymous to Boomi’s molecules that could be hosted on premise, on the cloud, or a mix. If you are interested more in Boomi, I have explained it previously in a similar article. You can have a look here
SnapLogic Main Components
On a high-level, SnapLogic consists of three main components as depicted below
Self-Service User Interface (IIP Portal)
This is effectively the control plane, where all the build, testing, administration, and monitoring occurs. It consists of three main sections which are easily viewable once logged in
SnapLogic Designer
The designer is an HTML-5 canvas where workflows are built in. Those workflows are called pipelines in SnapLogic terminology. The pipelines consist of collections of Snaps joined together to perform a single function. Snaps can be linked together using a straightforward “drag and drop” interface that does not require scripting.
SnapLogic Manager
The manager is the admin interface. This is where management, job organisation, project spaces, audit logs, permissions, user rights and alerts are defined and managed.
The manager can be accessed from the Manager tab at the top of the SnapLogic IIP interface.
SnapLogic Dashboard
This is where Snaplexes and integrations can be remotely monitored. The monitoring dashboard can be used on desktops, mobiles, tablets, among other devices. System health can be monitored in real time which allows operations teams to closely observe if there’s any anomalies in the system. Also, here is where you can view the history of executions in the platform and see all the logs and execution statuses
Snaplex
Snaplexes are one of the main foundational components of SnapLogic. As described in the architecture section, a Snaplex is the runtime engine over which the pipelines stream data between various application databases, files, applications, APIs, and big data sources. Depending on the amount of data being processed, the Snaplex can be defined to elastically scale using what is called by nodes. A node is a host over which a Snaplex is installed. So in short, a Snaplex can contain multiple nodes, and that’s how you scale and provide high availability for a Snaplex.
Again, a Snaplex can be deployed on the cloud (named a Cloudplex), on the client’s servers (named a Groundplex), as well as you can run it on Spark (named Hadooplex), and even on Raspberry Pi (named Raspberryplex). This shows how much this runtime is flexible and could be hosted in different environments.
Snaps
The last main component of SnapLogic is the snaps. These are nothing but the foundational activities that perform a single action. When grouped and connected together they form the pipeline. The “Snaps” feature of SnapLogic breaks-down the integration task complexity. The Snap can handle both straightforward as well as complex operations. It has the ability to carry out operations including reading data from a file, formatting data, transforming it, and writing it to some destination. As of now, there are more than 700+ different snaps in the platform which can virtually do anything. The amount of snaps that perform data manipulation and operations is nothing short of impressive.
Sample Snap: File Reader
SnapLogic Highlights
Here’s a quick highlights of platforms that differentiates it from other iPaaS platforms
- Simple and easy to use interface: SnapLogic is not the only simple one to use, it’s one of the simplest iPaaS platforms out there to use and build integrations.
- Iris AI: platform leverages a power AI system called Iris. The most notable feature here is suggesting the next snap whenever you drop in a snap to the designer canvas. This is based on all historical metadata stored in the platform and is claimed to be 90% reliable. Also, SnapLogic will soon release SnapGPT, a generative AI system based on ChatGPT that would allow you to create pipelines using natural language
- Real time data validation and preview: whenever you configure any snap, the platform in real time validates your work and retrieves a sample of the data that should go through the snap. This hugely improves the development experience and helps create integration pipelines that are bug free from the go.
PS: Whenever any snap is configured, it would automatically be validated and would turn green or red depending on if the validation is successful or not. Additionally, the dark green document icon at the right of the snap, upon being clicked, would show sample data that is coming out of the snap
- Data integration features such as ETL / ELT: there’s an impressive set of snaps that targets those scenarios such as loading, writing, merging, splitting, cleansing and formatting of data. This renders SnapLogic as one of the most powerful iPaaS platform for data integration
Conclusion
A product that helps integrate data efficiently and easily from everywhere has become essential in this age of digitisation, where businesses are going online and customers are using the internet to satisfy their numerous needs. This is when SnapLogic comes to the rescue. It aids companies in integrating cloud-based as well as enterprise platforms behind the company’s firewall.
The cloud-based platform is a blessing for many businesses since it satisfies all requirements of an iPaaS, as well as an ETL / ELT integration platform (application vs data integration requirements). The integration development work can be done accurately, quickly, and in a short amount of time, with the aid of AI. Hence, greatly improving developer efficiency and time-to-market. All-in-all, SnapLogic offers a comprehensive platform that fulfils various integration needs, assisting companies in integrating and automating their invaluable business processes.
Pingback: SnapLogic Step-by-Step Tutorial: Your First Integration