Our guest for today is David Reed, Lead Member of Technical Staff, Release Engineering at Salesforce.org. He’s very active as a moderator with some of the best answers I’ve seen on Salesforce Stack Exchange and has many talks and publications you can find here.
I work as a Salesforce Developer at EMPAUA and a moderator at SFXD. As a self-taught-admin-turned-developer, my mission is to interview the best people across our industry who helped and inspired me throughout the years and share what I have learned as I continue my journey in the Salesforce ecosystem. You can find my previous interviews and articles here. You can also find me on LinkedIn and Twitter.
Read More: Secrets to Developer Success – Interview with a Salesforce Technical Architect at Google
Atlas: Hey David, thanks for joining us today. To get us kicked off, can you tell us about yourself and how you got started in the Salesforce ecosystem?
David: I’ve been working in the Salesforce ecosystem for about six years now. I started my career as an admin-who-wears-many-hats in the nonprofit fundraising space, running Salesforce plus direct mail campaigns, managing events and gift processing. But I’ve been writing code since I was very young — my grandmother taught me BASIC when I was 9 or so, on a TI 99/4A that was probably older than I was — then I quickly taught myself Apex and moved into development. I worked as a developer, consultant, and architect in both the nonprofit and financial services sectors before I joined Salesforce.org in 2019. I like to claim (tongue somewhat in cheek) to be the shortest-tenured Salesforce MVP ever, as I joined Salesforce at essentially the same time as being elected.
Before starting to learn Salesforce, I was always that one person on a non-IT team writing tools to automate my job. I’d written a tool to generate printable catalogs while working as a late-night library supervisor, and on Raiser’s Edge, I built a primitive data loader and deduplicator. One of the things that excited me so much about moving to Salesforce was the open API and open ecosystem — the promise of being able to take that kind of tooling so much further.
My job today is in many ways my dream job. I get to design, build and apply more sophisticated tools, in Python, to make development on the Salesforce platform better.
I live in Denver, Colorado, USA. My non-technology interests include Ancient Greek (my academic background), botanical gardens, curling, and science fiction.
Open-source Tools Developed by Salesforce.org
Atlas: I’ve attended the great webinars that you and your team put together where an open-source suite of tools developed by Salesforce.org and the community are readily available for us to use. Can you tell us a little about these tools?
David: CumulusCI is the foundation of our suite. It’s a framework for defining automation on top of Salesforce DX, that makes it easy to build recipes for complex Salesforce orgs and workflows. Developers usually work with CumulusCI through the cci command-line interface, which lets you run automation to build, configure, and work with Salesforce orgs.
We build three web applications on top of the CumulusCI framework. MetaCI is our custom continuous integration harness that makes it easy to run CumulusCI automation in CI. (CumulusCI works with GitHub Actions, CircleCI, and others too; MetaCI helps with scaling and streamlining the whole process). MetaDeploy is our customer delivery platform — it powers install.salesforce.org, along with the installers for some other neat products (not all of which I can tell you about yet!). Customers bring their orgs to MetaDeploy to install our products and take advantage of our setup automation for a fast start.
Metecho is our new app that makes it easy for Salesforce admins and declarative developers to work in a modern development lifecycle based on scratch orgs and version control, without ever touching a command line. Metecho is a key player in the Salesforce.org Community Sprint program, where it’s being tested and honed.
Snowfakery is a tool that works with CumulusCI, for creating test data at scale. It’s amazing for seeding development and testing environments with complex, real-looking data that never risks personally identifiable information or production secrets, and it scales up to hundreds of millions of records easily for large data volume testing.
The things that I love most about working on this tooling suite is that it’s all 100% free and open source software. We build our tools collaboratively and with input from the community — including our internal product teams, ISVs, SI partners, and the incredible, generous people who participate in the Open Source Commons and Community Sprint programs. That genesis shows in the feedback we hear: that these tools are ready for building real products, in all of their complexity.
What Makes CumulusCI Unique?
Atlas: Can you tell us what makes CumulusCI unique among all the tools available in the industry? How did it come about and is it really free? What do you think about the state of DevOps in the ecosystem?
David: I think we’re at the start of a golden age in Salesforce DevOps. With so many offerings in the marketplace and so much knowledge growing in the community, there’s no excuse anymore for not having a DevOps strategy and not working in the modern world of source control and automated testing. And that’s a good thing for everyone, even if it can be a bit of an adjustment from older modes of working, because at the core, DevOps is about user trust: making sure we deliver high-quality software that meets our commitments.
CumulusCI is very different from other tools you’ll find, to the extent that I really don’t see it as a competitor to the major commercial packages. One key differentiator, of course, is that it’s free software, both in the sense that there’s no cost and that it’s available under the permissive BSD open source license.
In some ways, CumulusCI is opinionated. While you can use the tools with sandboxes, we heavily encourage using scratch orgs. Source control is not optional — we don’t offer a way to move metadata directly between orgs. We’ve baked in modern best practices and recommendations, but we also work hard to enable them and multiply their value.
In other ways, though, CumulusCI is all about your choices and the specific needs of your development process. You can use CumulusCI to build orgs to suit almost any need: install a suite of 11 managed packages, deploy some customizations, and then do a complex data load? That’s fine. Just configure an org setting and run a deployment? That’s fine too. Want to use our process for managing your branches and releasing package versions? Great! But if not, you don’t have to. You get the freedom to remix and assemble the tools we’ve built, or use the best practices we ship out of the box.
We’re also very intentional about building for the whole lifecycle. We want a CumulusCI project to offer automation that works for developers, for testers, for product managers, for release engineers, and for customers and partners. We know that model works because we practice it ourselves, in the open and at scale, where the whole community can observe.
The “Cumulus” in CumulusCI comes from the code name for the Nonprofit Success Pack (also proudly open source), which was the first large-scale application of our tools.
Atlas: Can you tell us how CumulusCI fits into the Package Development Model and Org Development Model?
David: We — that is, Salesforce.org — are what we call an “internal ISV”. We’re part of Salesforce, but we build and release managed packages just like other ISVs do, and we do so at a high scale. Last time I counted, we had over 40 active managed packages, and we ship new package versions every two weeks to our entire customer base.
Some of our products are very complex. The Nonprofit Success Pack, for example, includes six managed packages, additional unpackaged metadata, and setup automation as well. Other products, like Salesforce Advisor Link or Grants Management, ship with setup automation that does a full-on product implementation or builds and configures a Community for the customer, folding what would normally be a hundred-plus page setup guide into the product itself.
What I’m getting at here is that we’ve had to go a step above the Package Development Model in how we think about our products, to what we call the Product Delivery Model. The Product Delivery Model centers the notion of a product as shipped to a customer, no matter how complex. We use CumulusCI to implement the Product Delivery Model by building automation that builds and delivers the whole product: all of its packages, all of its unmanaged components, and the automated setup needed to make it usable and effective out of the box. That’s what our product is — not a single managed package, but everything that goes into that initial customer experience.
Core Capabilities of CumulusCI
Atlas: What are the core capabilities of CumulusCI? How does it compare to managing metadata effectively compared to standard tools?
David: CumulusCI does not compete with Salesforce DX — it complements DX. CumulusCI competes with two main opponents:
- Custom Bash and Node orchestration scripts that run a series of SFDX or Ant or API steps.
- Documents full of setup steps that users have to execute to get a working org.
We think CumulusCI offers a better alternative because it makes it easy to compose clear, reliable automation out of a massive library of built-in functionality. It abstracts away the low-level details and for most projects requires no code at all.
I like to describe CumulusCI as offering three major buckets of functionality….
- It gives you tools to create environments in which to do work. It manages dependencies effortlessly; it installs metadata, including situations where you may have different metadata for different types of orgs or workflows; it loads complex relational data; it configures org settings.
- It gives you tools to do work in your environment. For developers, it makes retrieving metadata and managing scratch orgs easy, and it has a clever Apex test runner that can retry common problems like row locks. For testers, it offers an amazing implementation of Robot Framework for browser-automation testing, and makes it easy to spin up a test org with a full set of data. For release engineers, it offers scaffolding for automated test builds and makes it easy to upload package releases.
- It offers tools for managing the whole application lifecycle. CumulusCI can help automate what we call CumulusCI Flow, a strategy for branch management in Salesforce projects; it can automate creating beta and released package versions, and deployments to sandbox and production orgs; it can create releases in GitHub and automatically generate release notes. And when coupled with MetaDeploy it can deliver the product directly to customers as well.
CumulusCI automation is built in YAML markup. For most projects, no code is required, although the capability to build your own tasks in Python is always available.
Run QA, Development, and Testing Scenarios for Different Sets of Users
Atlas: One thing I’ve noticed when going through the newly published documentation for CumulusCI, is that you can run automation for different sets of users and personas quickly with various scenarios QA, Development, and Testing. Can you tell us how we can do this?
David: I’m glad you got to use our new documentation — it was a major project that we’re really proud of!
“Automate everything” is a core tenet of ours, and it’s reflected in how we built CumulusCI. In a lot of traditional setups, automation runs in CI — but it doesn’t directly impact users’ workflows. We want CumulusCI to support, and offer out of the box, automation that directly enables not just developers, but also testers, product managers, support staff, demo engineers, partners, and community members.
We call the automation built with CumulusCI “portable automation” because it can be consumed in many different contexts. So, for example, you can define a sequence of steps that sets up an org for your product. Then, you can include that automation in flows that target different personas, like developers and testers — you generalize your base automation to serve everyone.
Then you can specialize further. Each of those flows (dev_org for developers, qa_org for testers, and so on) can have unique steps added that target the needs of a specific persona. A tester can start with a beta release of the application and then have a data load executed to get the org ready for their evaluation. A product manager persona can get an org set up to run a demo, tailored to the story you tell about your customers. Adding an element to a flow can be as little as two lines of markup.
What’s key is that all of those stories, those flows, are built out of the same automation building blocks, and they’re easy to extend, remix, and retarget as the product’s needs evolve.
Atlas: Is it possible to use CumulusCI with sandboxes and an org based development model? How does it compare to using the Product Delivery Model? Can we start using CumulusCI in an already existing org based on an org based development model and transfer it to the model you’ve described?
David: This is a great question and one that I hear a lot from folks in the community. I think it represents a desire to find that point where you get back the maximum impact at minimal effort applied to your infrastructure. In that light, I want to offer two answers.
First of all: yes, you can use CumulusCI with an org-based development model and with sandboxes. In fact, the final module of our Trail, Build Applications with CumulusCI, walks through how you might start an org implementation that’s built on top of the Nonprofit Success Pack. A CumulusCI-based development lifecycle usually ends in a persistent org — whether that’s packaging, for an ISV using first-generation packages, or a production Salesforce org, for an org implementation — but it’s also possible to build automation into the process to deploy against sandbox environments earlier in the lifecycle. CumulusCI automatically manages metadata deployment, installs new managed package versions, and handles component deletion for you in persistent orgs.
But I also want to make a case for the “big break” strategy. While CumulusCI can bring a lot of value to those org-based development workflows, I really believe that the greatest value is achieved by teams that invest the most heavily in automation and reduce the role of the sandbox — especially the persistent sandbox, that’s rarely or never refreshed — as much as possible. Being able to spin up a completely new environment at negligible cost is transformative.
Metadata ETL Functionalities of CumulusCI
Atlas: One feature I was really excited to see was the Metadata ETL functionalities of CumulusCI. Can you tell us about those?
David: Metadata ETL is also something that really excites me. We talked a little earlier about the Product Delivery Model, which embraces delivering large, complex products to customers that sometimes include unpackaged components and customizations. Practicing the Product Delivery Model leads us to think deeply about how to offer customers the ability to let us make changes to their orgs safely, to create a fully-rounded and integrated product experience with minimal manual setup.
Let me give you a concrete example. A package might have dependencies on specific values in the Case Reason standard picklist. You can’t deliver standard picklist changes in a managed package, so historically you’d have two options to deliver that package:
- Have the customer manually update the Case Reason picklist in every org in which they want to install. That’s not a great experience.
- Deliver the changes as an unpackaged metadata deployment, and risk overwriting other customizations made by the customer. That’s a potential trust violation!
What we do with Metadata ETL is extract the existing metadata that needs to be customized, make a targeted change to that metadata — like adding a specific new picklist value — while leaving the existing customizations unchanged, and then re-deploying the entire component.
It’s a great way to make surgical changes to metadata like Page Layouts, Global Value Sets, and even Org-Wide Defaults to safely achieve complex configuration in an org that may already be heavily customized.
I’ve used some ISV-specific language here, but I should also emphasize that the same paradigms apply well too, for example, the needs of departments that manage multiple Salesforce orgs, or consultancies delivering to their clients. Metadata ETL makes it safer to deliver new customizations in any situation where you don’t control the state of the target org.
Atlas: Another interesting feature that stands out is that we can pick and modify existing flows and automation that exists in CumulusCI and tailor those for specific use case scenarios. Can you tell us what’s needed to accomplish this?
David: Definitely! CumulusCI ships with lots of flows out of the box to build different kinds of orgs, like a dev_org for developers and a qa_org for testers. Projects can choose to use those flows unaltered, build their own flows, or deeply customize the out-of-the-box flows to suit their specific use cases.
Here’s one example with a bit of complexity to it. Say you’re a product tester and you’re working on a project that uses Experience Cloud. In order to do your job effectively, you need not just the project you’re working on in an org, but also a built out Experience, and a user you can log in as to evaluate how the project behaves. And ideally, you wouldn’t need to spend a half hour adding all of the data to make the org realistic.
So what you might do, working with your product team, is add several new steps to the flow config_qa, which configures a qa_org. One step would create an Experience (that’s an out-of-the-box task), another would create a User (consuming an SFDX command, or perhaps executing an Apex script), then a third loads up a data set that helps tell the story of the application you’re testing (that’s also out-of-the-box). It’s maybe 10 lines of markup, in total — no code — and then you’ve got a huge boost in your QA processes because you don’t have to set anything up by hand before you start your work in a new environment. In fact, this is representative of real automation we’ve implemented in partnership with development and QA teams for multiple projects.
What’s really cool is what happens when the rest of the team notices this shiny automation, and then you take the next step to factor that setup out into a flow you can use in lots of different ways: building product demos for business stakeholders, supporting developers, or perhaps even delivering product setup to customers through MetaDeploy.
What you should know before using DevOps tools
Atlas: It can be challenging for teams to adopt any tool and use it effectively. What should we have in mind before using any DevOps tool?
David: The key caveat in my mind is really simple: there’s no tool that solves 100% of your problem out of the box. (“Anyone who says differently is selling something”). There’s no tool and no process that in the long term is going to give you an out from investing in infrastructure or expertise or both, and it’s usually both. You’re always going to have to weigh tooling investments, people investments, and process investments together.
Another key factor to consider is that adopting DevOps successfully is likely to require you to confront unacknowledged weaknesses, and perhaps put some sacred cows out to pasture. Organizations that try to move to DevOps-based practice while clinging to the way their current processes work can hamstring themselves: they opt out of getting the most value from the best tools and process innovations because they fail to question and move past outdated assumptions. If you’re sitting on a mountain of unserviced or unacknowledged technical debt, that too can be a significant impediment (although once you make the switch, you may find that a serious DevOps program both helps you avoid accruing as much tech debt and helps you make the case to pay down what you have).
Online Communities
Atlas: David I know you are very active among online communities and your answers on StackExchange helps me out still to this day. For a professional at any skill level, can you tell us about the importance of participating in these, especially during COVID?
I also know you have joined SFXD – The Salesforce Discord Exchange recently, how was your experience?
David: I’m a huge fan of what online Salesforce communities can create — at all times, but all the more so during COVID, as we’re separated from our other communities and outlets for social connection, professional networking, and sharing of expertise.
All the communities (Salesforce Stack Exchange, SFXD, the community Slacks, Salesforce Twitter, and I’m sure others I don’t know yet) have different tones, foci and value to offer, so there’s really a niche for everyone.
One of the hardest things in community-building is setting goals, structures, and expectations, and I think that’s something SFXD and SFSE both do well (if not at all the same way). SFXD is a free-flowing, always-on discussion with a huge breadth of expertise. SFSE, where I’m a moderator, is a community knowledge base with a focus on building discoverable and rigorously structured solutions. I think that’s a great one-two; they serve different purposes and are both exceptional communities.
Coming out of my engagement with all of these communities, if there’s one piece of advice I can offer to engineers at all levels, it’s this: learn how to ask a good question. Be conscious and deliberate about sharing the right details and setting the right context to bring someone into the challenge you’re confronting so they can help you. Not only does that skill make you an asset to any online community you choose to join and get you to your solution faster, but it also helps you learn how to think through a problem effectively and makes you a stronger teammate.
That’s it for today’s episode! Thanks for tuning in, you can find me and David Reed at SFXD – Salesforce Discord Community Exchange and stay tuned for more!
Resources:
Build Applications with CumulusCI on Trailhead
CumulusCI On the Trailblazer Community
github.com/SalesforceFoundation/NPSP
github.com/SalesforceFoundation/EDA
Comments: