I’m a Solution Architect at ThirdEye Consulting, I’ve been working in the Salesforce ecosystem for nine years and I’m a certified Application and System Architect. I have been lucky enough to attend the System Architect Bootcamp at Dreamforce, which taught me a lot about integrations – which I had little experience of at the time. I have tried to present here an overview of what I learnt, but aimed at Salesforce professionals that are completely new to this.
My goal is not to teach you how to implement an integration step by step. Instead, I hope to give you an introduction to understanding integration concepts, ready for you to move on to the more complicated aspects later. There are three main topics to cover: integration types, capabilities, and patterns. But first, let’s take a closer look at the basics.
What is Salesforce Integration?
Salesforce Integration is the process of bringing two or more systems together, which allows you to streamline separate processes.
Think of cases in your own technology stack in which information is kept in one system but also required in another. By integrating them, you can easily manage all that information through multiple business processes that run across multiple systems.
Why is integration important?
We live in a digital era and we constantly need to improve our efficiency and our customers’ experience to remain competitive – it’s rare that a system works in isolation now. We need to become better and better at integrating our systems so that we can do it faster and in a scalable manner.
What is an API?
API stands for Application Programming Interface. It is the means that allows two applications to talk to each other.
For example, when you use an app on your phone, the app connects to the internet, grabs data from a server, and presents it back to you in a readable format. With the right API, this whole process should happen without a hitch.
There are different types of APIs which I will cover later in the ‘Salesforce Integration Capabilities’ section.
Types of Salesforce Integration Architectures
Each type of integration architecture offers both benefits and drawbacks worth noting. I will take you through the following three types:
- Point-to-point integration
- Hub-and-spoke integration
- Enterprise Service Bus integration (ESB)
Also known as one-to-one integration, a point-to-point system integration sends a message to another system via a 1-1 relationship. Imagine you have a sales application that sends new orders information to a billing system, to a shipping application, and to a tracking system – all separately as its own little integration. If you want the tracking and the shipping to communicate, then that’s another integration that needs to be set up.
There are lots of cons with this integration type. It’s expensive to build and maintain, and if you ever want to replace one of the systems, you will need to build multiple new integrations for it to connect to the same systems again.
With hub-and-spoke integration, you have a centralized hub system that facilitates communication between the systems. The hub is responsible for routing all traffic, and you only need to build one connection for each new system, which is a big improvement on point-to-point integration.
Enterprise Service Bus Integration (ESB)
Enterprise Service Bus (ESB) marks the evolution of the hub-and-spoke integration architecture model. The difference is the integration engine used to create connections between connected software applications and systems.
Use ESB if you need to do any of the following with your ‘messages’:
- Routing: if you want a ‘message’ to go from one system (A) to another system (B) or several systems.
- Orchestration: the order in which the transactions occur. For example, you might need a series of events to happen in a specific order. Imagine a scenario where one system needs to update something and give a response before that response can be used on a separate system.
- Transformation: if you want a ‘message’ to go from one system (A) to another system (B), but System B doesn’t speak the same language as System A – the bus first transforms that message into System B’s language.
- Security: it can provide authentication and authorization to meet security requirements.
Using an ESB, each of the systems within the integration gets an adapter that you can plug and unplug to your ‘bus’, making it more scalable as your integration requirements change over time.
Salesforce Integration Capabilities
As I write, I’m thinking: “how can I talk about APIs and not make it boring?” I’m going to keep this section short and ‘to the point’, but I hope it captures your interest and encourages you to learn more about APIs.
You need a high-functioning API that allows any two applications to communicate with each other. When we are talking about integrations, it’s important that we understand their timing and direction. Timing falls into two categories:
- Synchronous: You make a call to another system, but you have to wait for a response in return. The processing won’t continue until the response has been received.
- Asynchronous: You make a call to another integrated system, but you don’t have to wait for a response – further processing can continue. For example, if you have a background job that will take a long time to process, you don’t want to wait for the response from that to be created (as you risk a timeout).
The other consideration is the direction of the integration, which can be outbound or inbound. Ask the question of which system is the initiator; if the answer is Salesforce then it’s an outbound call, and if it’s another system, it’s an inbound call.
Integration capabilities are what Salesforce gives us in order to build these integrations. Here are eight that you should know about:
1. REST API
This focuses on data-based operations:
- GET to query a database
- POST to create a record
- PUT to update
REST API is best for web or mobile applications. Regarding data formats, it uses XML or JSON. JSON works better with data and is best for web or mobile applications. This means that REST doesn’t use much bandwidth and is easily consumed by web browsers.
In terms of timing, REST is synchronous. For example, you send a POST request to create a record in Salesforce and you will receive a response as to whether it was successful or not.
Salesforce Workbench is a suite of tools that allows you to interact with Salesforce using APIs. I recommend you do some exploring with Workbench to get familiar with the REST API.
2. SOAP API
SOAP API was more commonly used by older systems, but you may still come across it. It’s best for system-to-system integrations, back-end system communication, and for applications that require formal hand-off (contracts) between the API and the consumer (thanks to WSDL).
While it’s reliable and well-established, it tends to be slower and use more bandwidth than REST. It uses XML for very structured payloads and is asynchronous, meaning that it doesn’t need to wait for a response and can continue with other processing without causing blocks.
3. Bulk API
The Bulk API in Salesforce can handle high volumes of messages – up to 100 million records per rolling 24-hour timeframe. These APIs increase stability, allow you to monitor your jobs, and control your message loads.
It’s best suited for initial data migration as this option will allow you to load your organization’s data into Salesforce quickly. It allows you to control process jobs in serial mode (one after the other) or in parallel (multiple batches at the same time), and it’s clever enough to do automatic file batching.
As I’m sure you already guessed, it is asynchronous. You upload your load and then come back later to check the results.
4. Streaming API
There are two communication models: ‘Request and Response’ and ‘Event Driven’. The Streaming API is based on the publish/subscribe model, is a very efficient way to send notifications and is the basis of Event-Driven Architecture.
There are four Streaming APIs:
- Platform Events
- Change Data Capture
They are all built using CometD technology. This means there is an Enterprise Messaging Bus that allows an Event Producer to publish a Message to a channel so that any Event Consumer (subscribed to it) can pick up that broadcast. Platform Events simplify the process of communicating changes in data and allow us to have a near real-time integration between Salesforce and external systems.
There is a lot of buzz about Event-Driven Architecture at the moment and for good reason. I would encourage you to read more about it here.
5. Outbound Message
Whenever you have created a Salesforce workflow rule or approval process (if you have been using Salesforce for as long as I have!), I am sure you have seen the option to add ‘outbound message’ as an action. This is to send a message to another system whenever your event happens, and you can set this up declaratively!
Messages can be sent to any SOAP-based server endpoint that has a custom listener service. The message contains the fields you select when setting it up and is sent to your specified endpoint URL. This integration is asynchronous, but the integrated system will send back a message of acknowledgement. If Salesforce doesn’t receive the acknowledgment, it will continue retrying for 24 hours before notifying the System Admin user/s that the message was not successfully delivered.
Note that Outbound Messages are not designed to provide near real-time integration and Salesforce advises that the Streaming API should be used instead if that’s what you need.
6. Web Service Callouts
Callouts occur when Salesforce is calling out to another system (Salesforce is the ‘initiator’). For example, when a user updates an Account’s address field, Salesforce calls out to an address database to verify the new address (note: callouts are triggered by other events, not just field changes).
Web Service Callouts are:
- Free to use but are programmatic – you will need a developer to write the actions.
- Used when your timing requirements are synchronous, or you expect a quick response for asynchronous (note: callouts invoked using an Apex trigger must be tagged as asynchronous).
- Good if your data load is small (Request and Response < 3 MB).
7. Salesforce Connect
With Salesforce Connect, you can see and edit data from an external system in Salesforce. This is known as ‘Data virtualization’ – data is shown in Salesforce when required, but does not persist in Salesforce (we’ll revisit Data virtualization later). This is a declarative but paid feature. It does allow you to save up on data storage space because data is accessed in real time, and not stored in or copied into Salesforce.
How Salesforce Connect works:
- You create a new External Object (ending __x) that exposes the external data.
- It allows Lookups to other objects (e.g. you could have an Opportunity in Salesforce that displays Invoice and Invoice Line Items from an external billing system).
- Data is presented when called and you can run reports, create related lists, or run SOQL and SOSL.
- It can be used to connect Salesforce orgs.
8. Heroku Connect
Heroku is a cloud platform available as a platform-as-a-service (PaaS) that allows your business to develop, deliver, monitor, and scale various apps to bypass infrastructure obstacles. Heroku Connect is not Heroku itself (Heroku Postgres), it’s the connector between Salesforce and Postgres databases that enables ‘Data synchronization’:
When to use Heroku Connect:
Here’s an example of when you might need to use Heroku. During holiday promotions, a business wants to release a special app. They predict the app will receive millions of downloads, but they don’t want all that data in their Salesforce org – they only want the highlights. You can use Heroku Connect to get that data for you as well as sending updates from Salesforce. It allows for high volume with low latency and high throughput.
There are a few more capabilities for the UI layer, such as mashups and Canvas, but I’ll stop here for now.
Salesforce Integration Patterns
Patterns are abstract – they are just conceptual. Based on your integration requirements, you can identify the pattern and each of them has a host of best practices to help you identify what is the optimal capability for you to use.
This is when you need to think about the requirements around timing (synchronous or asynchronous), direction (inbound or outbound), and type. There are three approaches:
- Data integration: You’re integrating the information used by the systems, such as inserting data, updating data etc.
- Process integration: These are ways of integrating the process functionality against more than one system. For example, if you want to do billing automatically from an external system once an Opportunity is won in Salesforce – how do you pass that information to what needs to be billed and when?
- Virtual integration: Salesforce can interact with data that lives in another system without needing to replicate that in Salesforce. This is always triggered from something that happens in Salesforce, such as a user search or running a report.
Now that we covered that, here are the most common integration patterns that could meet your requirements.
Learn more: Salesforce’s Pattern Selection Matrix is a great resource.
The data stored in Salesforce is created, retrieved, updated, or deleted by a remote system. For example, when a business uses an order system and needs to update Salesforce with the order status.
Request and Reply
Salesforce invokes a process on a remote system and waits on completion of that process (synchronous) to get a response (for example, Callouts).
Fire and Forget
Salesforce invokes a remote system process but does not wait for completion. Instead, the remote system acknowledges the request and then hands off control back to Salesforce – hence firing, forgetting, and resuming activities.
Examples of Fire and Forget integrations:
- Outbound messages
- Platform events
Batch Data Synchronization
Data stored in Salesforce is created or refreshed to reflect updates from an external system. Then changes from Salesforce are also sent to an external system on a defined schedule. Updates in either direction are done in a batch manner. For example, you can extract customer activity information from Salesforce and import it into an on-premise data warehouse weekly.
UI Update Based on Data Changes
The Salesforce user interface (UI) needs to be updated as a result of changes to Salesforce data – for example, Case Manager needs to see Case Update Status. You need to create a custom user interface and use the Streaming API with PushTopic that you can subscribe to and see the alerts.
Data virtualization is when data is shown in Salesforce when required, but does not persist in Salesforce. Data virtualization came up earlier when we covered Salesforce Connect, but it can also be used for Apex SOAP, or REST Callout via a Visualforce page.
As you can see, Salesforce Integration can be broken down into many parts. The first step is ensuring that you understand what integration is and why it’s important.
This introduction to integration concepts has highlighted the different types of integration architectures, capabilities, and patterns. It’s well worth exploring these concepts to discover what Salesforce integration services are required for your specific org and business users.
If you want to learn more, Integration Patterns and Practices is the best resource I’ve seen to date.