I spend a great deal of time dwelling on what makes a good Data Migration.
I identified the six attributes of a good data migration, which form the foundation of my book “Developing Data Migrations and Integrations with Salesforce: Patterns and Best Practices”. These characteristics are what I call the “why” behind the best practices that I discuss later in the book.
The principle attribute is “well planned”, for one simple reason:
Failure to meet any of the other five attributes can be mitigated by planning, but nothing can mitigate for a bad plan—or worse, no plan.
This post will cover the basics of data migration planning – what good planning looks like, the essentials questions to ask, planning for risks and how your project manager is a valuable asset in your data migration process. Data migration is a topic that’s often daunting, risky and with conflicting advice online, which is why I hope to use my experience to make the subject clearer.
What does good planning look like?
Your migration plan should look like a traditional project plan, with timelines, dependencies, and milestones. It’s likely your project manager (PM) will need your help in building this plan, unless they have the technical knowledge. I suggest you work with your PM to create a proper plan – you are the migration specialist, after all! Then they can take care of tracking your progress as the migration commences.
What are the essential questions to ask at the start?
Your plan should answer each of the following questions. If you don’t have an answer at that moment, provide a deadline of when the question will be answered:
- How and when will the data to be migrated be delivered (by the legacy system owners)?
- What format will the data be delivered in?
- Who is the source data subject matter expert (SME)?
- Are there any data security concerns or guidelines for handling the data?
- What middleware or ETL tool will you use to code your migration?
- How much time is required for data analysis?
- How clean is the data? Is a data cleanup mini-project wise? If so, who is responsible, when will it be done, and in what system?
- When will the data mapping document be completed and who is responsible for creating it?
- When will you review the data mapping document with your client?
- What Salesforce environment will be used for quality assurance (QA), User acceptance testing (UAT), and the Production Migration? If the target Salesforce Org is currently being built out, when will the various objects be available so that you can load data to them?
- When will you perform the first complete end-to-end run of the migration?
- Who will be performing QA / UAT? How many rounds for each?
- When do you go live?
- How much time do you have for the production cut over? (This may be downtime for users; they may not be able to use the legacy systems or Salesforce.)
This list is a good basis, but doesn’t even touch on the technical side of planning (eg. setting up environments, gathering transformation rules, selecting tools, and so on).
Every migration is different, and time needs to be spent on planning and ensuring all your bases are covered. You don’t want surprises late in the game, answer these questions early.
Plan for an extended UAT Period
When I ask users to perform UAT, I ask them to test for ‘Intent.
What is intent? In this context, I mean the business goals.
It’s common for code, including data migrations, to meet the technical specified requirements, but completely miss the ball with regard to business intent.
Unfortunately, intent defects are a time-consuming process. Once logged, they are likely to require a meeting to confirm the issue is real, and what should be done about it. Therefore, you should plan for users to have enough time to thoroughly test the data. Additionally, UAT testers are often performing UAT in addition to their day job, you can’t expect them to spend 4 hours a day testing, every day for 2 weeks. Make sure you understand their schedule and they understand the commitment needed to properly perform UAT.
Your Project Manager is your friend and ally, work with them.
When working with your project manager, make sure they understand your dependencies and you understand the Salesforce build plan. Once you do, work with your PM to assemble a detailed schedule that includes all the dependencies.
This means that when the time comes to start coding the migration of a particular object:
- The Salesforce build is complete for that object
- It has been deployed to your target environment
- You have received the data from your client
- You have enough time to analyze it and code all the transformations.
For QA, UAT, and production migration runs, make sure the dates land on days that you can dedicate the time needed for the jobs to run.
Your PM is your friend and ally, if your client or the Salesforce development team is not providing you with what you need to do your job, your PM can step in and hold them accountable. However, PMs can do this only if they have a solid plan that includes the detailed list of exactly what you need and when.
I cannot tell you how often I come across data migrations where the plan was to “get .csv files from the client and then load them in.” This simply does not meet the requirements of a good plan, not even for a “small” data migration. Not even close.
You must have a plan and execute that plan. Plan your data migration like a project in its own right. Track your risks and mitigate them with even more planning. Plan for failure; plan for bad requirements; plan for clients who are nonresponsive or indecisive, or who simply give you bad information (unintentionally or even intentionally). Just have a good detailed plan!
This article is adapted from my book “Developing Data Migrations and Integrations with Salesforce: Patterns and Best Practices” (Apress December 2018), available to buy on Amazon.