A New Approach to Predictive AI in Salesforce Using OpenAI Assistants and ML

In this article, we’ll be exploring innovative alternatives for integrating predictive AI and machine learning (ML) into the Salesforce platform. This post will introduce a novel approach that leverages machine learning models and OpenAI Assistants, trained on Salesforce data, to enhance predictive capabilities and insights for users. When available, I typically like to start with a business case to ground us in the “why?”.

You are a Salesforce Architect working for a company that is curious about AI capabilities within the Salesforce ecosystem. You have been meeting with several business units, capturing pain points, and understanding growth initiatives that would serve as potential use cases for AI-driven solutions. During a meeting with stakeholders, you listen as Sales reps discuss the prioritization of opportunities and their lack of insights into which are most lucrative to pursue. The wheels start to turn and you begin to think about the variety of AI options that you could bring to Salesforce to provide helpful insights for their daily decision-making.

Below you’ll see a visual preview of what we will unpack in this article. Everything you see in the screenshot was created using native Apex, a Lightning Web Component, Salesforce data, a machine learning algorithm, and OpenAI Assistants. If this intrigues you, read on!

Let’s Rewind

Traditionally, Salesforce users have looked to Einstein Analytics as the go-to solution for embedding intelligence into their CRM activities. While extremely powerful, Einstein’s capabilities come with their own set of challenges, particularly around cost and extensibility, which can be prohibitive for many organizations.

Furthermore, the quest for tailored machine learning solutions that can seamlessly integrate with Salesforce data and processes often leads businesses down a path of complex, resource-intensive custom development projects.

Salesforce offers a tremendous amount of predictive capabilities within their Einstein ecosystem and actually served as the inspiration for this idea. You can learn all about them with a trail called Get Smart with Einstein.

As a starting point, I spun up a free Salesforce org with Analytics Studio installed and built a binary classification model with Einstein Discovery trained on Opportunity data. A binary classification model predicts one of two possible outcomes, such as “win” or “lose” in this context. The purpose of this model is to use features from the Account and Opportunity object such as Industry, Lead Source, Amount, and Type and predict if the Opportunity will win or not. 

Here is a small sample of the dataset:

Above is the dashboard that Einstein Discovery provides for you after your model is trained. Among the many great insights and features this dashboard provides, I will focus on the ROC AUC score of .8183 in the “Review Model Accuracy” card.

The Area Under the Curve (AUC) is a key metric in machine learning that measures a binary classifier’s ability to distinguish between classes. An AUC of 1 indicates perfect prediction, while an AUC of 0.5 suggests no better than random chance. Essentially, the higher the AUC, the better the model at correctly classifying positive and negative outcomes. Additionally, an AUC that is extremely close to 1 could also indicate overfitting a model. For more explanation on AUC, read this article.

Now that the model is built, we can drag it onto an Opportunity record to get insights. Here is a capture below of the out-of-the-box card for the prediction data and insights on my Opportunity that the model produced:

After building this, I started to wonder if there was another way to unlock these predictive capabilities for organizations looking to grow and scale with deeply meaningful insights.

As beneficial as Einstein can be through its predictive AI tooling, what if your organization can’t afford the licensing? Or as beneficial as a custom machine learning solution that’s integrated into Salesforce is, what if your organization’s enterprise isn’t mature enough to take that on?

So, what other options do you have when traditional paths are either too costly or complex?

Enter Open AI Assistants

Separately, I have recently been exploring Open AI Assistants for a few other use cases and the more I got to know the capability, the more ideas started to swirl. Here is a high-level diagram of how Assistants with Open AI work:

The Assistants API allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries. The Assistants API currently supports three types of tools: Code Interpreter, Retrieval, and Function calling.

Code Interpreter: allows the Assistants API to write and run Python code in a sandboxed execution environment. 

Retrieval: augments the Assistant with knowledge from outside its model, such as proprietary product information or documents provided by your users.

Functions: allows you to describe functions to the Assistants and have it intelligently return the functions that need to be called along with their arguments. 

For this example, we will be using the code interpreter tool in our Assistant. Now, let’s break down the anatomy of Assistants with Open AI and how we will use each mechanism to accomplish our goal.

Side note: While it’s common for Assistants to be built and configured programmatically, I will show examples through the Open AI Assistants playground which, as you’ll see, actually acts as a cool configuration tool that impacts how we will interact with our Assistant programmatically.

Assistants

Assistant: represents an entity that can be configured to respond to users’ messages using several parameters like:

• Instructions: how the Assistant and model should behave or respond.
• Model: you can specify any GPT-3.5 or GPT-4 models. The Retrieval tool requires at least gpt-3.5-turbo-1106 (newer versions are supported) or gpt-4-turbo-preview models.
• Tools: the API supports Code Interpreter and Retrieval that are built and hosted by OpenAI.
• Functions: the API allows you to define custom function signatures, with similar behavior as our function calling feature.

Here are the instructions for the Opportunity Prediction Assistant that we will create:

You are a bot that uses machine learning and opportunity data sets from Salesforce to predict if the opportunity will win or lose based on the Opportunity Type, the Lead Source, the Amount, and the Industry. You will use the machine learning code provided to you, which is a gradient boosting classifier machine learning algorithm, and the csv dataset to predict the Is_Won dependent variable for new prediction requests that come in.

In the files section of the assistant builder, I have added two files to provide the assistant with the context required to perform an action. I’m not sure if “simple” and “machine learning” ever belong in the same sentence, but the first Python file is a simple GradientBoostingClassifier machine learning algorithm that I wrote that will be trained on our Opportunity dataset and enable opportunity predictions. See the code below:

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

from sklearn.metrics import roc_auc_score
import pandas as pd
# Load the dataset
df = pd.read_csv('/mnt/data/opportunity_history.csv')


# Preprocessing
X = df.drop('IsWon', axis=1)
y = df['IsWon']


# Splitting the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)


# Preprocessing for numerical and categorical data
numeric_features = ['Amount']
categorical_features = ['Industry', 'Lead Source', 'Opportunity Type']
numeric_transformer = Pipeline(steps=[
   ('scaler', StandardScaler())
])
categorical_transformer = Pipeline(steps=[
   ('onehot', OneHotEncoder(handle_unknown='ignore'))
])
preprocessor = ColumnTransformer(
   transformers=[
       ('num', numeric_transformer, numeric_features),
       ('cat', categorical_transformer, categorical_features)
   ])
# Gradient Boosting Classifier
clf_gbc = Pipeline(steps=[('preprocessor', preprocessor),
                         ('classifier', GradientBoostingClassifier())])


# Training the model
clf_gbc.fit(X_train, y_train)


# Predicting and evaluating the model
y_pred_gbc = clf_gbc.predict_proba(X_test)[:, 1]




# Get probability of the positive class
auc_score_gbc = roc_auc_score(y_test, y_pred_gbc)

How did I land on using a GradientBoostingClassifier machine learning algorithm for these predictions? I had it compete with other classifier models to see which had the best outcome:

• Logistic Regression
  • Accuracy: 70.67%
  • ROC-AUC Score: 75.12%
• Random Forest Classifier:
  • Accuracy: 71.93%
  • ROC-AUC Score: 74.89%
• Gradient Boosting Classifier:
  • Accuracy: 77.75%
  • ROC-AUC Score: 82.15%

What Is a Gradient Boosting Classifier Model?

Imagine you’re trying to solve a complex puzzle. You start with a guess, but quickly realize it’s not right. Instead of starting over, you focus on where you went wrong and make a small adjustment. You repeat this process, each time getting closer to the solution. This is essentially how a Gradient Boosting Classifier works, but instead of solving puzzles, it’s making predictions based on data!

Now that we have our model and our prepared dataset uploaded, we are ready to interact with our Assistant and fine-tune it to our needs.

Threads and Messages

Next, we will create our thread and add relevant messages. Threads and messages represent a conversation session between an Assistant and a user. Conceptually, this isn’t that different from your own usage of Chat GPT. You navigate to Chat GPT (your assistant) and create a new chat (thread) and you start engaging with the assistant with specific requested prompts.

Let’s walk through a few messages. Immediately below, you can see that I am providing the assistant with further instructions and specific context:

At the risk of boring you, what is not represented here is the Assistant’s amazing ability to process your request and work through any issues it encounters. As it parses the Python file, checks the code, processes the code, and analyzes the dataset, the Assistant is working on all of this in real-time and self-correcting; it’s pretty amazing to watch.

Fast forwarding, the Assistant has finished consuming the machine learning code, training the model, and has provided an ROC AUC score for the model, indicating that this specific model has a good predictive performance.

There are two extremely important takeaways from this:

• Firstly, I want to acknowledge that we just created a machine learning algorithm and used a code interpreter Assistant to train our model on a dataset that we provided, enabling us to make real-time predictions… this is mind-blowing to me!
• Second, the comparison of our GradientBoosting model’s ROC AUC score to the Einstein Discovery model we built earlier. Granted, there are so many elements in machine learning that go into the accuracy and efficacy of a model, but the fact that we are able to build a model with a better AUC Score is extremely inspiring and exciting.

So, with the assistant now using our machine learning model and dataset for Opportunity predictions, how do we use the assistant in Salesforce?

When we think about interacting with the assistant programmatically, it’s very similar to how we were interacting in the playground above. We create a thread and prompt the assistant with a message and the assistant responds. Let’s walk through the diagram above and the steps that were taken:

1. Assistant Batch: The assistant batch job is designed to create a new assistant thread, fetch all qualified opportunities that need an IsWon prediction, and use grounded Opportunity data and one-shot prompting as context to generate the assistant message. We then run the assistant thread.

2. Creating a thread:

curl https://api.openai.com/v1/threads \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -H "OpenAI-Beta: assistants=v1" \
 -d ''

This request creates a thread for your message.

3. Creating a thread message:

curl https://api.openai.com/v1/threads/thread_abc123/messages \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -H "OpenAI-Beta: assistants=v1" \
 -d '{
     "role": "user",
     "content": "How does AI work? Explain it in simple terms."
   }'

The image above uses sample content. However, I have added my actual message content below. As you can see, I am providing the assistant with a JSON example of what to include in the response and the format as well.

'{\n' +
'  "prediction": "win or loss",\n' +
'  "win_probability": "probability percentage",\n' +
'  "comparative_insights": [\n' + // Changed to an array
'   {\n' +
'   "type": "amount",\n' + // Added type field
'   "description": "Insightful comparison"\n' + // Renamed to description
'   },\n' +
'   {\n' +
'   "type": "lead_source",\n' + // Added type field
'   "description": "Insightful comparison"\n' + // Renamed to description
'   },\n' +
'   {\n' +
'   "type": "industry",\n' + // Added type field
'   "description": "Insightful comparison"\n' + // Renamed to description
'   }\n' +
'  ],\n' +
'  "recommendations": [\n' +
'   {\n' +
'   "feature": "Specific feature",\n' +
'   "recommendation": "Specific actionable recommendation"\n' +
'   }\n' +
'  ],\n' +
'  "dataset_overview": {\n' +
'   "average_win_amount": "numeric value",\n' +
'   "most_successful_industry": "industry name",\n' +
'   "least_successful_industry": "industry name",\n' +
'   "best_lead_source": "source name",\n' +
'   "opportunity_type_with_highest_win_rate": "type name",\n' +
'   "average_lead_source_effectiveness": {\n' +
'   "Source name": "effectiveness percentage",\n' +
'   "Another source": "effectiveness percentage"\n' +
'   }\n' +
'  }\n' +
'}';

One thing I want to point out is that the first two (prediction and win_probability) are results of the actual machine learning algorithm that we built into the assistant earlier. The remainder of this is added information and insights about the dataset that I am asking the assistant to put together in real time. 

I think this is pretty incredible. Not only can we use assistants to provide predictions, but we can also have them tell us meaningful information about our data which indicates a high level of potential. 

4. Running a thread:

curl https://api.openai.com/v1/threads/thread_abc123/runs \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -H "Content-Type: application/json" \
 -H "OpenAI-Beta: assistants=v1" \
 -d '{
   "assistant_id": "asst_abc123"
 }'

The last part of this batch actually runs the thread for the message. This basically means that you are now allowing the assistant to process your message and provide a response to you. 

5. Status Batch: The status batch job is designed to periodically check on the status of the thread run. Thread runs are asynchronous which is understandable as we don’t really know how long it will take for an assistant to respond to your thread run. So, the batch job runs at a fixed interval to check if a thread run has been completed. Once the status batch gets a completed thread run status, it retrieves the assistant response and updates the Opportunity record.

Below is a diagram detailing the run lifecycle:

Using Generative AI With Grounded Predictive Data

When visualizing this data in a custom LWC on the Opportunity record page, there are two ideas I had.

The first is to create a component with strict mappings to the JSON attributes that appear in the expected JSON output of the assistant. This would allow us to have a ton of control over how we want to visualize each piece of the data that is returned to us and create a good-looking component for our users to see. This should look familiar from the beginning:

The other idea was to make the component more modular and use the <lightning-formatted-rich-text/> lightning web component to show the formatted rich text of our prediction insights that we get back from the assistant. The title of the component is just a configurable attribute in the app builder. I decided to build a callout to the ChatCompletion OpenAI API and provide it with this prompt below:

You are a helpful bot that formats prediction JSON data into a single line of rich text suitable for insertion into a `<lightning-formatted-rich-text/>` component in Salesforce LWC. The prediction result (“win” or “lose”) should be emphasized with a larger font size and color-coded (green for “win”, red for “lose”). 

Include `<br/>` tags to insert new lines at the start of major sections and use bullets for sub-items under each section. Ensure that each element within the “Dataset Overview” is individually bulleted. The formatted text should maintain high readability and clear structure across sections including “Comparative Insights,” “Recommendations,” and “Dataset Overview.”

Below are the nine lines of code it took to visualize the data you see in the capture below:

<template>
   <div>
       <lightning-card icon-name="utility:einstein" title="Predict Win / Loss">
       <p class="slds-p-left_medium">
           <lightning-formatted-rich-text value={prediction}></lightning-formatted-rich-text>
       </p>
       </lightning-card>
   </div>
</template>

Granted, the capture above is not as good as the explicit JSON attribute mapping component from the first idea. However, I think it’s a decent option for the scalability you would get out of generating rich text from a ChatCompletion call and your ability to drop this component anywhere you want.

OpenAI Code Interpreter Pricing

The Code Interpreter is billed at $0.03 per session. If your Assistant invokes the Code Interpreter in two separate threads simultaneously – for example, one thread for each end-user – then two distinct Code Interpreter sessions are initiated. Each session remains active by default for one hour. Therefore, you are charged for only one session per thread if users engage with the Code Interpreter within the same thread for up to an hour. Reference OpenAI Pricing for more information. 

This price is really appealing, especially if we strategically manage thread usage within the same sessions that multiple users are leveraging for real-time or batch predictions. 

Important Considerations

Production readiness: OpenAI Assistants, as of writing this, are still in beta. Details of future releases and production support continue to come in and this could have implications on long-term AI strategies and technology usage.

ML Ops: There is a significant question around best practices and standards when it comes to deploying and leveraging machine learning models in this way. More research and testing will be required to determine if leveraging code interpreter technology in an assistant is a suitable way to leverage machine learning technology.

Consistency of Assistant Responses: In this example, we are telling the assistant how to respond to us and in what format we would like the response to be in. It is not guaranteed that this will be the case every time, however, we can write excellent prompts to ensure that it returns to the format we wish most of the time. We also have other options such as the new “response_format” of “json_object” that you can add to OpenAI calls now which could help ensure a more consistent response. 

Team skills: A large assumption with this approach is that you have people in your organization who can understand a use case, write necessary machine learning code, and maintain it over time. 

Summary

AI Assistants offer tremendous potential that extends well beyond code interpretation and machine learning applications. These tools open up a realm of possibilities previously inaccessible, presenting diverse use cases that can revolutionize how we utilize AI in Salesforce. 

What do you think? Could this approach realistically achieve predictive AI on the Salesforce platform and serve as a bridge between Einstein’s capabilities and enterprise maturity?

Let’s keep innovating and find out the answers to these questions!