Teach an Old Org New Tricks – enabling Scratch Orgs in legacy project

Theory

Think of Scratch Orgs as fresh, temporary Salesforce instances you can spin up on demand. They are used primarily by developers to build and test new features, configurations, or packages in an isolated environment that can be easily created and discarded.

Why you want to use Scratch Orgs

There are many factors that might make you want them:

your boss returning from a Salesforce conference
you want to get rid of outdated developer sandboxes
creating a safe playground that can be easily destroyed and re-created
developers overwriting each other’s work on the same dev environment
waiting weeks for the admin team to refresh your sandbox
and more…

But hey, not all that glitters is gold. Scratch Orgs are great for greenfield projects, but who works in greenfield?

Enabling Scratch Orgs in a legacy project with a lot of technical debt sounds like a challenge. Grab my hand, and I will show you how deep the rabbit hole goes.

This is not going to be a straightforward task, and there is no unique recipe, as many features and settings do not work correctly with Scratch Orgs. So, prepare for many story points, coffee, or other drinks for this journey.
I am going to explain, based on my experience, how I approached this task. It may not be ideal, but it worked, so it can’t be that bad, right? Right!?

Salesforce Limits and Workarounds

Salesforce has a limit of 10,000 components being deployed at once. The same applies to retrieving metadata.

This is why you will probably need to split the deployment into small parts and deploy them one by one.

In most cases, you’ll want to deploy managed packages first, then your custom components.

My journey and lessons learned

What did I want to achieve?

A simple Scratch Org with installed managed packages and the full repository deployed. No generated/seeded data yet; let’s start simple.

Journey beginning

So, I enabled Dev Hub in production, and I was able to create a scratch org.
We were already using DX/source format, so I didn’t need to migrate anything.
The project-scratch-def.json was a default one; no extra settings or features were added.

Managed Packages

I listed all installed managed packages in Production and installed them in the Scratch Org using the Package Version ID. If you have multiple dependent managed packages, the order of installation is crucial; the error messages will guide you on what should be installed first.
Once I had all packages installed, it was time for our custom code!

Metadata Deployments

Let’s start with deploying the foundations, so Custom Objects and related items.

Oh boy! Multiple wild errors appeared.

Most of them were due to a missing dependency because a previous metadata deployment failed, so that’s not too bad. But after studying some of them, it turned out that I was missing many features and settings in my Scratch Org.

After a couple of rounds of creating a scratch org, installing packages, and ~~deploying metadata~~ (oh sorry – attempting to deploy metadata), I gave up, as there were hundreds of these features and settings used, and I had no time to waste.

Scratch Org Shape

Scratch Org Shape to the rescue!
What is it? I would say it’s just a shell with features and settings (most of them) from Production that you can use instead of creating project-scratch-def.json from scratch. That’s all – it doesn’t have metadata or data.
But it also contains licenses associated with managed packages, so if the exact same version of a package is installed in the org shape destination, you can install it in the scratch org without knowing the installation key for the managed package. Be aware that if the version changes on the target org shape, you cannot install the old package version without an installation key. I really recommend using the --installation-key parameter, even though you can skip it when creating scripts, as ISVs can perform push upgrades, the package version changes, and your automation crashes.

I didn’t want to use org shape in the first place, as I remembered it had some limitations and not all settings/features were supported. But after I gave it a try and RTFM, there are only a few things that need to be manually added to project-scratch-def.json.

What’s not included in the Scratch Org Shape?
Metadata API settings with integer or string fields and some exceptions – you will find examples in the docs.

So I enabled Org Shape in Dev Hub, created an org shape via SF CLI, and included it in the scratch org definition file.

The number of errors decreased from 1000+ to circa 900.

Split metadata into parts

So now the fun part starts: A deep dive into errors, figuring out:

what should be deployed first
what needs recalculation after deployment so we can start another deployment
what requires enabling some features and settings

Eventually, I was able to split it into portions.

Here are 2 examples of splitting metadata. Remember, this probably won’t work for your project without adjustment; this is just to show you the idea and the approach.

Example 1

Part 1 – non-dependent metadata

authproviders
brandingSets
cachePartitions
certs
contentassets
cspTrustedSites
customPermissions
dataSources
groups
labels
LiveChatButtons
LiveChatDeployments
managedContentTypes
messageChannels
mlDomains
namedCredentials
notificationtypes
remoteSiteSettings
roles
settings
Skills
staticresources

Part 2 – Foundations

aura
classes
components
customMetadata
email
flexipages
flows
globalValueSets
letterhead
lwc
objects
pages
quickActions
standardValueSets
tabs
workflows
wrappers

Part 3 – the rest with dependencies

audience
dashboards
EmbeddedServiceConfig
experiences
navigationMenus
networks
reportTypes
reports
sharingRules
sharingSets
sites
testSuites

Example 2

Part 1

cachePartitions
contentassets
cspTrustedSites
customPermissions
globalValueSets
groups
labels
remoteSiteSettings
roles
standardValueSets
staticresources
translations

Part 2

customMetadata
documents
email
letterhead
objects
queues
settings

Part 3

applications
aura
classes
components
dw
emailservices
flexipages
flows
homePageComponents
homePageLayouts
lwc
pages
pathAssistants
quickActions
sharingRules
tabs
triggers
weblinks
workflows

Part 4

approvalProcesses
assignmentRules
dashboard
dashboards
duplicateRules
escalationRules
fieldRestrictionRules
layouts
matchingRules
objectTranslations
permissionsetgroups
permissionsets
profilePasswordPolicies
profileSessionSettings
profiles
reportTypes
reports

I haven’t sorted everything upfront, but I was able to, piece by piece, deploy/validate some metadata types and divide them into parts.

Also, you might need to move some dependencies between parts – if webLinks (part 2) which are part of objects have references to a Visualforce page (part 3), then you will have to move just webLinks to be deployed along with pages (part 3).

Org-specific items

So now, after I had sorted the dependencies, the focus was on removing org-specific items like usernames. In some projects, that may be as easy as just replacing OrgWideEmailAddress with CurrentUser and removing senderAddresses in workflows. Other projects are way more complex, especially if you want to deploy profiles and permission sets with Managed Package item access. Oh dear, that’s really tedious and frustrating work.

Why is that? Sometimes when you have your managed package upgraded in Production, even though the latest package version does not contain certain components like Visualforce pages or Apex classes, these are not removed from your org.
That leads to a misalignment between a new environment, such as a scratch org where only the new package is installed, and an old environment with extra deprecated metadata.

So now you can imagine how many unnecessary files Production can have. If you store the full profile in the repository, you have to take that into consideration and change profiles/permsets with some automated scripts before you can deploy to a scratch org.

Or you can just remove managed package access from profiles/permission sets. Or even better, have as little access in profiles as possible and use permission sets and permission set groups. That should solve this problem too. Eventually, you will just skip the deployment of a few permission sets; that’s way easier than dynamically modifying huge XML files with profiles.

Let’s not forget about Org-specific items in Profiles, like access to Dev Hub, or access to features which were not reproduced in the Scratch org for some reason – these also have to be handled.

What are the other customizations needed for a Scratch Org?
Once again, here is a list of possible customizations:

User/Email Replacement:

Replace hardcoded usernames with the current scratch org username and remove duplicated usernames, if any.
Update queue members to a single current user.

Settings Change:

Enable Chatter and Enhanced Notes.
Disable MFA opt-in.
Remove Case routing addresses.

File/Folder Deletion:

Remove specific folders (objects, object translations, etc.).
- Remember to delete references of deleted objects/fields from other metadata like profiles/permission sets/layouts.
Delete specific files (layouts, tabs, permission sets).
Remove unsupported/unneeded metadata (i.e., topicsForObjects, samlssoconfigs, certs).
Delete email templates if you have more than 500.
All limits which were increased by a support case might now be an issue; try to delete non-essential items in that case.

Other Component Modification:

Replace OrgWideEmailAddress with CurrentUser, and remove senderAddresses in workflows.
Remove Visualforce/Flexipage action overrides.
Clean up flexipages (remove items without values).
Update layouts (button replacements, remove specific sections).
Clean profiles/permission sets (remove specific permissions).
Rename listView labels if they contain too many characters (worked in Prod, but not in Scratch).
Change encryptionScheme in fields to None.
Limit lookupPhoneDialogsAdditionalFields in the object definition file to 10, as limits on scratch orgs are different than in Production.

Data issues

At some point during deployment, you might also hit errors due to missing data – a good example would be missing currencies.
You can export the data to a JSON file from Production:

sf data export tree --query "SELECT IsoCode, ConversionRate, DecimalPlaces, IsActive, IsCorporate FROM CurrencyType" --json --output-dir data --target-org Production

and then import it into the scratch org using:

sf data import tree --files data/CurrencyType.json --target-org ScratchOrg

MVP is done

After a long battle with Scratch Orgs, I finally managed to have a script that creates a scratch org and deploys most of the metadata from Production.

It takes a lot of time, as there are many managed packages and many deployments to perform, but hey, it works, and it is always better to see a progress bar than manually preparing an environment.

So, all that hard work to get basically what you could have when creating a new developer sandbox?

Well, yes and no. This was just the MVP; scratch orgs have more potential than sandboxes when it comes to development.

Possible next steps are:

Let developers use them instead of using outdated developer sandboxes.
Enable Scratch Org Pools and prepare the scratch orgs during the night.
If there is always a big queue of Pull Request validations with unit tests to some non-production sandbox, we can offload the traffic to scratch orgs and deploy metadata with unit tests there – this will be our validation job, and we can run them in parallel.
Prepare fake data scripts in the Scratch org using Snowfakery + CumulusCI.
And more, as there are a lot of possibilities to improve the process with ephemeral environments.

Thank you for being with me on that journey. Below you will find the checklist for enabling scratch orgs in your organization, just in case someone has the crazy idea of enabling that in a legacy project.

Good Luck!

Scratch Orgs Checklist

Enable Dev Hub:
Use your actual production org, or a dedicated org, as the Dev Hub.
Setup > Development > Dev Hub -> Enable

Also, I suggest enabling Org Shape under Setup > Development > Scratch Orgs -> Enable
Install Salesforce CLI
Follow the instructions: https://developer.salesforce.com/docs/atlas.en-us.sfdx_setup.meta/sfdx_setup/sfdx_setup_install_cli.htm
Also, make sure you have authorized with the Dev Hub:
sf org login web --set-default-dev-hub --alias dev-hub
DX Project Structure and Source Format
Migrate from MDAPI to DX if you haven’t already:
https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_source_file_format.htm

Create Org Shape
Using this command:

sf org create shape --target-org Production

Update the Scratch Org Definition file with the ID of sourceOrg (the ID of your Production org used as –target-org):

{
"orgName": "Beyond The Cloud Scratch Org",
"sourceOrg": "00DB1230000Ifx5",
"features": ["Communities", "ServiceCloud", "Chatbot"],
"settings": {
    "communitiesSettings": {
        "enableNetworksEnabled": true
    },
    "mobileSettings": {
        "enableS1EncryptedStoragePref2": true
    },
    "omniChannelSettings": {
        "enableOmniChannel": true
    },
    "caseSettings": {
        "systemUserEmail": "contact@beyondthecloud.dev"
    }
}
}

Create Scratch Org and Deploy Metadata
Also, install managed packages.
Make sure you have noted all modifications and steps you have taken to deploy everything for the first time.
You will have to recreate everything using a script.
Put everything into a script.
It will break.
Going with the above approach, at some point, the script will surely fail. It can be a Salesforce Major Release which changed something, or you may hit the limit of either deployment (10k) or a particular metadata part. Just accept that and keep in mind to monitor the creation of scratch orgs so you can be faster than an annoyed developer reporting that your automation is not working again.

References: