Which stage of the indexing pipeline divides text into tokens?

The indexing pipeline in Oracle Text processes text for search:

Correct Answer (D): “Lexer” divides text into tokens (words, symbols) based on language rules and settings (e.g., whitespace, punctuation). It’s the stage responsible for tokenization in Oracle’s text indexing process.

Incorrect Options:

A: Sectioner identifies document sections (e.g., headers), not tokens.

B: Tokenizer is a generic term, but in Oracle Text, “Lexer” is the specific component.