Apex Queueable Processing Framework

Intro

Did you ever run into "Too many queueable jobs added to the Apex job queue" errors, due to multiple queueable entrypoints?

Or maybe even worse, you tried to overcome this by using @future methods, only to find out that they are limited to 50 calls per transaction and the same governor limits apply?

And what then? Did you try to use long if (System.isFuture() && System.isBatch() && System.isQueueable() && ...) statement to not execute the asynchronous code when already in asynchronous context?
But why? Probably it should be executed anyway, right? So this is more like a workaround than a solution.

If you answered "yes" to any of the above questions, then this guide is for you!

Problem Statement

Imagine that you created new Queueable job to run some logic asynchronously.

public class MyQueueableJob implements Queueable {
    public void execute(QueueableContext context) {
        // Your logic here
    }
}

You are happy with it, so you test it and it works as expected.

Month later, your collegue comes to you and says that he created another Queueable job that does some other logic, but for some reason it is throwing an "Too many queueable jobs added to the Apex job queue" error, even though he only enqueued one job in his Queueable job.

public class MyColleagueQueueableJob implements Queueable {
    public void execute(QueueableContext context) {
        // Your logic here
        // Enqueue another job from queueable job
        System.enqueueJob(new MyColleagueQueueableJob());
    }
}

After some investigation, you both find out that in the MyColleagueQueueableJob there is an operation chain that is executing the same logic where your MyQueueableJob is also enqueued, and that is causing the "Too many queueable jobs added to the Apex job queue" error, since both jobs are being enqueued in one Queueable context transaction.

Since both jobs should be executed, you try to find a way to overcome this issue. You start looking for some workarounds to enqueue the job from different contexts, such as @future methods, batch jobs, or even scheduled jobs. Your colleague even suggests to skip the enqueueing of the job when already in a queueable context, but you know that this is not a good solution, as it will not execute the logic when it should, and this can happen again later with his job too.

Knowing that in future there will be more entrypoints for asynchronous processing, you start looking for a solution that will allow you and your team to enqueue queueable jobs in a managed way, without worrying about the limits and entrypoints.

And this is where the AsyncLib framework comes in handy!

What is this guide about?

This guide will state the limits of the Apex queueable processing and how to overcome them by using a queuable chains. There will be also a few examples of the AsyncLib framework, which is a library that helps to build such chains in a simple way.

Let’s dive!

Salesforce Limits

The Salesforce limits for executing/enqueuing asynchronous jobs/methods from different contexts are as follows:

*Although scheduled Apex is an asynchronous feature, synchronous limits apply to scheduled Apex jobs

That means that if you are in a synchronous context, you can enqueue up to 50 queueable jobs, 50 future methods, and up to 100 batch jobs in the holding status. And if you are in a queueable job context, you can enqueue only 1 queueable job and 50 future methods.

Importantly, the total number of Apex jobs in the queue is limited to 250,000 per 24 hours. This includes all types of asynchronous jobs, such as @future, queueable, batch, and scheduled jobs.

How to overcome the limits in managed way?

As we can see, the limits for queueable jobs are quite strict, especially when it comes to chaining them. The only way to overcome this is to manage the queueable jobs in a way that allows you to enqueue only one job at a time, while still being able to execute multiple tasks asynchronously.

But how to do that when in the system there are multiple entrypoints, such as @future methods, queueable jobs, and batch jobs, and we know there would be more in future?

The answer is to use a queueable chain. A queueable chain is a way to enqueue a queueable job that will execute another queueable job after it finishes. Since we got Finalizer interface from Salesforce, that allows us to execute some code after the queueable job finishes (regardless of job success/failure), we can use it to enqueue another queueable job in a chain in a managed way. Here is more about the Finalizer interface.

How to implement a queueable chain?

To implement a queuable chain we need:

1. An abstract Apex class that implements the Queueable interface
2. An Apex class that implements the Finalizer interface
3. An Apex class for managing the queueable chain
4. An Apex class that will be the entrypoint for the queueable chain
5. Mechanism for enqueueing all the jobs in the chain in the initial transaction
6. Mechanism of tracking the queueable job results from the chain
7. A way to handle errors and retries in the queueable chain

Seems like a lot of work, right? But don’t worry, there is already a solution for that!

We can use the AsyncLib framework, which provides a simple way to create and manage queueable chains. The framework allows you to define a chain of queueable jobs that will be executed one after another, while also allowing you to handle errors and retries.

With the AsyncLib framework, the framework takes care of the queueable chain management, so you don’t have to worry about the implementation details.

But how it works in practice? Let’s dive into usage examples.

Usage examples

Queueable

1. Create a class that extends QueueableJob.

// QueueableJob class example
public class MyQueueableJob extends QueueableJob {
    public override void work() {
        // To access the current job context
        Async.QueueableJobContext ctx = Async.getQueueableJobContext();
        // Your logic here
    }
}

2. Enqueue the job using the Async.queueable() method.

// QueueableJob enqueue example
Async.queueable(new MyQueueableJob())
    .enqueue();

3. Optionally, you can set various properties on the job, such as delay, priority, asyncOptions and behaviour on error.

// QueueableJob enqueue example with options
Async.queueable(new MyQueueableJob())
    .delay(10) // Delay in minutes
    .priority(10) // Set job priority, lower number means higher priority
    .asyncOptions(new AsyncOptions()) // Set async options
    .continueOnJobEnqueueFail() // Continue on job enqueue failure
    .continueOnJobExecuteFail() // Continue on job execution failure
    .rollbackOnJobExecuteFail() // Rollback on job execution failure
    .enqueue();

4. To access the current job context, use Async.getQueueableJobContext() within the work() method of your job class.

// Accessing the current job context
Async.QueueableJobContext ctx = Async.getQueueableJobContext();
ctx.currentJob; // The current QueueableJob instance
ctx.queueableCtx; // The QueueableContext instance
ctx.finalizerCtx; // The FinalizerContext instance

5. If you need to handle finalization logic, you can implement the FinalizerContext in your job class.

// FinalizerContext example
public class MyQueueableJobFinalizer extends QueueableJob.Finalizer {
    public override void work() {
        // Finalization logic here
        Async.QueueableJobContext ctx = Async.getQueueableJobContext();
        ctx.finalizerCtx; // Access the finalizer context
        // Access the job context and perform finalization
    }
}

6. Attach the finalizer job in the QueueableJob context.

// Enqueueing a finalizer job
Async.queueable(new MyQueueableJobFinalizer())
    .attachFinalizer();

Batchable

1. Create a standard Batchable class that implements Database.Batchable.
2. Use the Async.batchable() method to execute the batch job.

// Batch execute example
Async.batchable(new MyBatchableJob())
    .execute();

3. Optionally, you can set scopeSize or minutesFromNow, but the latter can be used only when sheduling.

// Batch execute example with scope size
Async.batchable(new MyBatchableJob())
    .scopeSize(100) // Set the scope size for the batch job
    .execute();

// Batch schedule example with minutesFromNow
Async.batchable(new MyBatchableJob())
    .minutesFromNow(10) // Schedule the batch job to run in 10 minutes
    .asSchedulable() // Convert BatchableJob to Schedulable
    .name('My Scheduled Batch Job') // Set a name for the scheduled job - required
    // No CRON expression set since we are using minutesFromNow
    .schedule();

Schedulable

1. Use Batchable or QueueableJob class or create a standard Schedulable class that implements Schedulable.
2. If using Batchable or QueueableJob, ensure it uses asSchedulable() method to convert it into a Schedulable job.

// Batchable Job as Schedulable example
Async.batchable(new MyBatchableJob())
    ... // further configuration of batchable job
    .asSchedulable() // Convert Batchable Job to Schedulable
    ... // further configuration of scheduled job

// Queueable Job as Schedulable example
Async.queueable(new MyQueueableJob())
    ... // further configuration of queeuable job
    .asSchedulable() // Convert Queueable Job to Schedulable
    ... // further configuration of scheduled job

3. If using the Schedulable job use Async.schedulable() method to schedule the job.

// Schedulable execute example
Async.schedulable(new MySchedulableJob())
    ... // further configuration of scheduled job

4. Configure the job parameters

// Schedulable execute example with parameters
Async.schedulable(new MySchedulableJob())
    .name('My Scheduled Job') // Set a name for the scheduled job - required
    .cronExpression('0 0 12 * * ?') // Set a cron expression for scheduling - required
    .schedule();

5. Set Cron expression to be either a valid cron expression or a valid cron builder instance.

// Example of using CronBuilder via String
.cronExpression('0 0 12 * * ?')
// Example of using CronBuilder
.cronExpression(
    new CronBuilder()
        .buildForEveryXMinutes(10) // Build a cron expression to run every 10 minutes
)
// Example of using CronBuilder with a specific time
.cronExpression(
    new CronBuilder()
        .buildForEveryXHours(5, 25) // Build a cron expression to run every 5 hours starting at 25 minutes past the hour
)

Additional features

The AsyncLib framework also provides additional features such as:

Async Result – Custom Object

Custom Object AsyncResult__c is created for each QueueableJob enabled in configuration that is processed. It contains the following fields:

• CustomJobId__c: The custom job ID generated by the framework.
• SalesforceJobId__c: The Salesforce job ID of the processed job. This can be used to check the job details directly in Salesforce.
• Result__c: The status of the job (SUCCESS, UNHANDLED_EXCEPTION).

Queueable Job Settings – Custom Metadata Type

Custom Metadata Type QueueableJobSettings__mdt is used to configure the Queueable Job settings. It contains the following fields:

• QueueableJobName__c: The name of the QueueableJob.
• IsDisabled__c: Boolean field to indicate if the QueueableJob is disabled.
• CreateResult__c: Boolean field to indicate if the AsyncResult should be created for the job.

There is a default Queueable Job Settings record called All created in the package, which can be used to configure the default settings for all QueueableJobs.

But how is it built? Let’s dive into framework components.

Components of the AsyncLib framework

• Async: Entrypoint class for all the async operations. It provides methods for enqueueing queueable jobs, executing batches, and scheduling jobs.

Queueable

The AsyncLib framework consists of several components that work together to create and manage queueable chains. Here are the main components:

• QueueableBuilder: Builder class for creating queueable jobs. It allows you to define the job parameters and the finalizer that will be executed after the job finishes.
• QueueableManager: Class for managing the queueable jobs. It keeps the singleton instance of the queueable chain and checks if the normal queueable job should be executed or if it should be added to the queueable chain based on current limits.
• QueueableChain: The heart of the Queueable processing. It executes current job, passes conexts and handles errors.
• QueueableJob: Abstract class that implements the Queueable interface. It is used to create queueable jobs that can be executed in the queueable chain.
  • This class allow to show the respective enqueud job in Flex Queue, not a generic QueueableChain, what helps with easier understanding about job results and monitoring
• QueueableChainFinalizer: Finalizer class that is executed after the queueable job finishes. It allows you to enqueue another queueable job from the chain.
  • _This is the only finalizer that is used, and the ones added by MARKDOWN_HASHc4a67464b37d4030aa1dde5e991abbdcMARKDOWNHASH are treated as chained Queueable Job with higher priority.
• QueueableChainBatch: Batch class used to enqueue the chain job in the initial transaction due to limits.
  • Why do we even need this?
• QueueableSchedulable: Class that allows you to wrap a queueable job into a schedulable job, so it can be scheduled.

Queueable Job – Apex Class

// QueueableJob defnition
public abstract class QueueableJob implements Queueable, Comparable {
    // The Salesforce job ID of the processed job
    public Id salesforceJobId;
    // The custom job ID generated by the framework
    public String customJobId;
    // The name of the class that implements QueueableJob
    public String className;
    // A unique name for the job, used to identify the job in the queue
    public String uniqueName;
    // Delay in minutes before the job is executed
    public Integer delay;
    // Priority of the job, lower number means higher priority
    public Integer priority;
    // Options for the job execution
    public AsyncOptions asyncOptions;
    // Flag to indicate if the job has been processed
    public Boolean isProcessed = false;
    // Flag to continue on job enqueue failure
    public Boolean continueOnJobEnqueueFail = false;
    // Flag to continue on job execution failure
    public Boolean continueOnJobExecuteFail = false;
    // Flag to rollback on job execution failure
    public Boolean rollbackOnJobExecuteFail = false;
    // The QueueableContext instance for the job
    public QueueableContext queueableCtx;

    // The custom job ID of the parent job, used for finalizers
    public String parentCustomJobId;
    // The FinalizerContext instance for the job
    public FinalizerContext finalizerCtx;
    // Flag to indicate if the job is a finalizer
    public Boolean isFinalizer {
        get {
            return String.isNotBlank(parentCustomJobId);
        }
    }
    ...
}

Queueable Job Context – Context Wrapper

// QueueableJobContext definition
public class QueueableJobContext {
    // The current QueueableJob instance being processed
    public QueueableJob currentJob;
    // The QueueableContext instance for the current job
    public QueueableContext queueableCtx;
    // The FinalizerContext instance for the current job, if applicable
    public FinalizerContext finalizerCtx;
}

Async Result – Result Wrapper

// AsyncResult definition
public class AsyncResult {
    // The Salesforce job ID of the processed job
    public Id salesforceJobId;
    // The custom job ID generated by the framework
    public String customJobId;
    // Flag to indicate if the job is part of a chain
    public Boolean isChained;
}

Batchable

The AsyncLib framework also provides a batchable component that allows you to execute batch jobs in the queueable chain. The batchable component consists of:

• BatchableBuilder: Builder class for creating batch jobs. It allows you to define the job parameters such scope or delay.
• BatchableManager: Class for managing the batch jobs.
• BatchableSchedulable: Class that allows you to wrap a batchable job into a schedulable job, so it can be scheduled.

Schedulable

The AsyncLib framework also provides a schedulable component that allows you to schedule jobs in the queueable chain. The schedulable component consists of:

• SchedulableBuilder: Builder class for creating schedulable jobs. It allows you to define the parameters such name, and cron expression. Also any Queueable and Batcheable classes can be wrapped into the Schedulable job and sheduled with ease.
• SchedulableManager: Class for managing the schedulable jobs.
• CronBuilder: Class for building cron expressions. It allows you to define the cron expression in a simple way.

Explanations for curious minds

Why not use @future methods?

The @future methods are hard to maintain due to only option to pass primitive types as parameters. It is better to use Queueable jobs anytime. Here is the nice comparison of the @future methods and Queueable jobs.

How finalizers work in AsyncLib?

The finalizers in the AsyncLib framework are used to execute code after the queueable job finishes. They are implemented as a separate class that extends the QueueableJob.Finalizer class. The finalizer is attached to the queueable job using the attachFinalizer() method.

These jobs are treated as chained Queueable Job with higher priority. Since finalizers has the same limits as synchronous processes, with the exception of 1 enqueued job, 50 future methods and 12MB of heap size, this approach is only allowing for more flexibility with attached finalizers.

Why do we need to use QueueableChainBatch?

To be efficient, in synchronous context we are trying to enqueue as many queueable jobs as possible. That means, once reaching 50 queueable jobs, we need to enqueue the rest of them using the QueueableChain.

The issue is that we don’t know how many jobs still will be enqueued in transaction over the limit, and System.enqueueJob() is passing only current state of the Queueable Job. That means, if we would enqueue the QueueableJob with chain details as a 50 job, and later there would be additional one, adding it to the chain will not work, since the chain was already enqueued.

During testing it was found that using System.enqueueJob() and System.abortJob() in the same transaction is still using the limits of Queueable Jobs… That means, it is not possible to push any number of jobs to the chain.

On the other hand, using Database.executeBatch() is not tied to the same limits, and when using System.abortJob() it will return to the previous limits. That means we can execute this job with the current state of the Queueable Chain, and if there is a new job to enqueue, the batch will be aborted and the new one will be executed instead. This way we can enqueue as many jobs as possible in the initial transaction, and then continue with the chain execution.

Considerations

• If using AsyncLib framework, all the jobs should be enqueued using the Async class methods, such as Async.queueable(), Async.batchable(), or Async.schedulable(). This ensures that the jobs are properly managed and executed in the queueable chain.
• All the @future methods should be replaced with Queueable jobs, as they are more flexible and easier to maintain.
• If attaching QueueableJob.Finalizer, they are using one additional execution limit, since they are executed after the main job finishes, not as a real finalizer.
• Because of the atomicity of transaction, when QueueableChainBatch is executed and aborted in the same transaction, it will still be visible as aborted job in the Flex Queue. This is expected behavior, as the job was executed and then aborted to execute the next one. The number of aborted jobs will be equal to the number of initial jobs enqueued in the transaction.
• Due to possiblity that there could be jobs added to the chain but not enqueud from Salesforce perspecitve, some of them will not return salesforceJobId in the AsyncResult wrapper. This is expected behavior, as the job was not enqueued in the initial transaction, but was added to the chain. The customJobId will still be available for tracking purposes via AsyncResult__c custom object if needed.
• When sheduling jobs, the returned value is List<AsyncResult> instead of AsyncResult, as there could be multiple jobs scheduled in the same transaction, since Salesforce does not allow to schedule jobs to run more than one per hour, so to achieve that we need to schedule multiple jobs in the same transaction with different minute offsets to achieve it.

Outro

As you can see, there is some work that needs to be done to implement a queueable chain in a managed way. But with the AsyncLib framework, you can easily create and manage queueable chains without worrying about the implementation details.
If you have any questions or suggestions, feel free to open an issue on the AsyncLib.