For cost-effectiveness and high availability in Amazon EMR workloads, the best approach is to configure atransient cluster(which runs for the duration of the job and then terminates) withOn-Demand Instancesfor the primary and core nodes, andSpot Instancesfor the task nodes. Here's why:
Primary and core nodes on On-Demand Instances: These nodes are critical because they manage the cluster and store data on HDFS. Running them on On-Demand Instances ensures stability and that no data is lost, as Spot Instances can be interrupted.
Task nodes on Spot Instances: Task nodes handle additional processing and can be used with Spot Instances to reduce costs. Spot Instances are much cheaper but can be interrupted, which is fine for non-critical tasks as the framework can handle retries.
Atransient clusteris more cost-effective than a long-running cluster for workloads that only run for 6 hours a day. Transient clusters automatically terminate after the workload completes, saving costs by not keeping the cluster running when it's not needed.
Option A: A long-running cluster may result in unnecessary costs when the cluster isn't being used.
Option C: Running core nodes on Spot Instances risks data loss if the Spot Instances are interrupted, violating the requirement for zero data loss.
Option D: Running both core and task nodes on Spot Instances is highly risky for data-critical workloads.
AWS References:
Amazon EMR Cluster Management
Using Spot Instances in EMR
