A mule application is required to periodically process large data set from a back-end database...

When a Mule application uses batch job scope to process large datasets and is deployed on multiple CloudHub workers without persistence queues enabled, the following scenario occurs if a worker crashes:

Batch Job Scope: Batch jobs are designed to handle large datasets by splitting the work into records and processing them in parallel.

Non-Persistent Queues: When persistence is not enabled, the state of the batch processing is not stored persistently. This means that if a worker crashes, the state of the in-progress batch job is lost.

Worker Crash Consequence:

When a worker crashes, the records that were being processed by that worker are not tracked persistently.

As a result, when another worker (or the same worker after a restart) picks up the job, it does not have the previous state information.

This leads to the batch job starting from the beginning and reprocessing all records from scratch, causing duplicate processing of records that were already processed before the crash.

This behavior can cause issues such as duplicate data in Salesforce CRM and inefficiencies in processing.

References

MuleSoft Batch Processing

MuleSoft CloudHub Workers