You have an Azure Databricks workspace that contains an all-purpose cluster named Cluster1.

Diagnosing OOM errors requires analysing actual runtime execution behaviour. The Spark UI is the primary tool — it's accessible from the cluster detail page and captures rich per-stage and per-task metrics without any extra setup.

In the Executors tab, look at storage memory used, execution memory used, memory spill to disk, and GC time per executor. An executor showing high memory spill is a strong indicator that it's processing more data than it can hold in memory — often caused by data skew, where one partition is far larger than the others.

The Stages tab shows task distribution — if one task in a stage is processing 10x more data than its peers, that's data skew causing memory pressure on that specific executor. Ganglia (available on older runtimes) provides node-level OS metrics like heap usage over time, which can confirm whether memory pressure is sustained or spiky.

These built-in tools give a complete picture of the root cause before making any configuration changes.