A company wants to build a dimension table in an Amazon S3 bucket.

Option D provides the lowest runtime because it uses a table format designed for efficient incremental upserts on Amazon S3, rather than repeatedly scanning and rewriting large portions of a 1 TB dataset. Although Spark on its own (Options A and C) can perform joins/merges, updating files stored in S3 typically requires expensive rewrites, especially as data grows. By contrast, Apache Hudi is purpose-built for maintaining large datasets on object storage with incremental updates, which directly fits “update up to 10,000 records every day” without reprocessing the full historical footprint.

For the compute layer, the document highlights that Amazon EMR provides a fully managed environment for running Apache Spark and other big data frameworks to process and analyze large datasets, making it appropriate for high-scale processing where performance matters. This is a better fit than using Pandas on 1 TB (Option B), which is not designed for distributed processing at that scale.

Therefore, combining EMR + Spark with an incremental storage framework (Hudi) is the most runtime-efficient approach for daily record-level updates on S3.