Which concept refers to breaking text into smaller units for processing by LLMs?

Tokenizationis the foundational process by which an LLM breaks down raw text into smaller, manageable units called "tokens." These tokens can represent individual words, parts of words (sub-words), or even punctuation marks. This is a critical step because LLMs do not "read" words like humans do; they process numerical representations of these tokens. The way text is tokenized directly impacts the model's efficiency and its ability to understand complex technical terminology used in software testing. For example, a rare technical term might be broken into several sub-word tokens. This process is closely linked to theContext Window(Option C), which is the maximum number of tokens a model can "remember" or process at one time. WhileEmbeddings(Option B) are the numerical vectors that represent the meaning of these tokens, and theTransformer(Option A) is the underlying architecture that processes them, tokenization is the specific mechanism for initial text decomposition. Understanding tokenization is vital for testers when managing long requirement documents to ensure they do not exceed the model's limits.