What is a prerequisite to define a custom pattern for a repository?

You must enable secret scanning before defining custom patterns. Secret scanning provides the foundational capability for detecting exposed credentials, and custom patterns build upon that by allowing organizations to specify their own regex-based patterns for secrets unique to their environment.

Without enabling secret scanning, GitHub will not process or apply custom patterns.