An engineer has two DataFrames: df1 (small) and df2 (large).

Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Question Answer

An engineer has two DataFrames: df1 (small) and df2 (large). A broadcast join is used:

python

CopyEdit

frompyspark.sql.functionsimportbroadcast

result = df2.join(broadcast(df1), on='id', how='inner')

What is the purpose of using broadcast() in this scenario?

Options:

It filters the id values before performing the join.

It increases the partition size for df1 and df2.

It reduces the number of shuffle operations by replicating the smaller DataFrame to all nodes.

It ensures that the join happens only when the id values are identical.

Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 PDF/Engine

Get 75% Discount on All Products, Use Coupon: "ac75sure"