Which approach demonstrates a modular and testable way to use DataFrame.

Databricks and Apache Spark recommend building modular and reusable ETL transformations by leveraging the DataFrame.transform() API. This method allows you to chain multiple transformation functions in a clean and testable way.

Option A : Encapsulating the logic in a class (Pipeline) works, but it reduces modularity and flexibility. It does not show the true intended use of DataFrame.transform() which is chaining functional transformations.

Option B : This is the correct approach. It defines small, reusable functions (upper_value, filter_positive) that each take a DataFrame and return a transformed DataFrame. By chaining them with df.transform(func), you can compose ETL pipelines in a clear and declarative manner. This enables unit testing of individual functions and makes the ETL pipeline modular, testable, and production-ready.

Option C : This shows a single transformation wrapped in a function and tested, but it lacks pipeline composition — it is not demonstrating modular chaining across multiple transformations.

Option D : This simply defines a transformation function with hardcoded logic. It does not leverage DataFrame.transform() nor demonstrate modularity through composition.

Therefore, Option B is the best demonstration of how to use DataFrame.transform() in PySpark ETL pipelines.

Databricks documentation explicitly highlights that DataFrame.transform() allows developers to “chain together reusable functions in a readable and modular way, improving testability and maintainability of ETL code.” This makes B the correct and officially supported pattern.