A data engineering team runs nightly batch ETL pipelines that transform large volumes of data...

A job compute cluster triggered by a scheduled workflow is appropriate for predictable nightly ETL. Job compute is designed for non-interactive workloads and is created for the job run, then terminated automatically when the run finishes. It can be configured with autoscaling so that worker capacity adjusts to the workload, while the organization pays for compute only during execution rather than maintaining an always-running cluster. An all-purpose cluster is optimized for interactive development and can incur idle costs. A single-node cluster cannot scale across workers and is unsuitable for large-volume processing. A serverless SQL warehouse is intended for SQL workloads, whereas the scenario describes a general Delta Lake ETL pipeline. Consequently, option D provides the required scheduling, scaling, reliable isolation, and cost efficiency.

================