You are creating an Apache Spark job in Azure Databricks that will ingest JSON-formatted data.

Convert nested JSON to a flattened DataFrame

You can to flatten nested JSON, using only $"column.*" and explode methods.

Note: Extract and flatten

Use $"column.*" and explode methods to flatten the struct and array types before displaying the flattened DataFrame.

Scala

display(DF.select($"id" as "main_id",$"name",$"batters",$"ppu",explode($"topping")) // Exploding the topping column using explode as it is an array type

withColumn("topping_id",$"col.id") // Extracting topping_id from col using DOT form

withColumn("topping_type",$"col.type") // Extracting topping_tytpe from col using DOT form

drop($"col")

select($"*",$"batters.*") // Flattened the struct type batters tto array type which is batter

drop($"batters")

select($"*",explode($"batter"))

drop($"batter")

withColumn("batter_id",$"col.id") // Extracting batter_id from col using DOT form

withColumn("battter_type",$"col.type") // Extracting battter_type from col using DOT form

drop($"col")

)