In Spark’s distributed architecture:
The Driver Node coordinates the execution of a Spark application. It converts the logical plan into a physical plan of stages and tasks.
The Executors, running on Worker Nodes, are responsible for executing tasks assigned by the driver and storing data (in memory or disk) during execution.
Key point:
Executors are the active agents that perform the actual computations on data partitions. Each executor runs multiple tasks in parallel using available CPU cores.
Why the other options are incorrect:
A (Driver Nodes): The driver schedules tasks; it doesn’t execute them.
C (CPU Cores): CPU cores execute within executors, but they are hardware, not Spark architectural components.
D (Worker Nodes): Worker nodes host executors but do not directly execute tasks; executors do.
References (Databricks Apache Spark 3.5 – Python / Study Guide):
Spark Architecture Components — Driver, Executors, Cluster Manager, Worker Nodes.
Databricks Exam Guide (June 2025): Section “Apache Spark Architecture and Components” — describes the roles of driver and executor nodes in distributed processing.
