A company has developed a large language model (LLM) and wants to make the LLM...

Use Case 1:

The company’s chatbot needs predictions from the LLM to understand users ' intent with minimal latency.

→ Answer: Real-time inference

Chatbots require low-latency, immediate responses to user input. Real-time inference is ideal for these interactive use cases.

Use Case 2:

A data processing job needs to query the LLM to process gigabytes of text files on weekends.

→ Answer: Batch transform

Batch transform is designed for asynchronous, high-throughput jobs where latency is not critical. It’s ideal for scheduled or large-scale processing like weekend batch jobs on large datasets.

Use Case 3:

The company’s engineering team needs to create an API that can process small pieces of text content and provide low-latency predictions.

→ Answer: Real-time inference

An API requiring fast response time for small content sizes is best served by real-time inference to meet latency requirements.