A transformer-based model is the best answer because modern real-time conversational speech foundation models use transformer architectures to understand context, reason over user input, and generate suitable conversational responses. AWS provides a direct example through Amazon Nova Sonic and Amazon Nova 2 Sonic, which are designed for low-latency speech-to-speech interaction.
AWS describes the architecture of Nova Sonic as combining specialized speech components with a multimodal LLM. The AWS AI Service Card states: “we trained a core transformer model on a variety of multilingual and multimodal data sources.” AWS also describes Nova Sonic as a speech-to-speech model that supports natural, real-time voice conversations with low latency.
A complete speech-to-speech system may contain speech encoders and speech renderers or decoders, but neither an encoder-only nor decoder-only model adequately represents the complete conversational architecture required by the scenario. An encoder converts an input such as speech into internal representations, while a decoder generates outputs. A bidirectional conversational system must perform both understanding and generation while maintaining conversational context.
Diffusion models are most strongly associated with iterative generative processes such as image generation and some audio-generation workloads. They are not the best architectural category for the low-latency, contextual, real-time conversational agent described here.
AWS Nova Sonic demonstrates the intended pattern particularly well: speech is accepted as input, contextual reasoning occurs through a multimodal transformer-based LLM, and speech is produced as output through a bidirectional streaming interface. This enables interactive voice assistants, customer-service agents, and similar real-time applications.
Therefore, among the available options, transformer-based is the technically correct model type.
===========
