A penetration tester is conducting an external assessment of a corporate web server.

The CEH web server attack methodology describes reconnaissance as a key phase, where testers gather publicly available information before attempting exploitation. Robots.txt is commonly used by administrators to instruct web crawlers about which directories should not be indexed. CEH emphasizes that attackers regularly review robots.txt because it often exposes sensitive directories unintentionally, providing valuable intelligence about internal structure, configuration paths, administrative pages, and potential weak points. In this scenario, the tester observes “Disallow” entries and then discovers the directories are not protected by authentication, allowing direct access to sensitive files. This falls under information gathering through exposed indexing instructions rather than directory traversal, which involves path-manipulation exploits. The tester is not altering file paths or inserting traversal sequences; instead, they are reviewing publicly available indexing instructions and discovering misconfigured access controls. This perfectly aligns with the reconnaissance phase of the CEH methodology, where attackers learn about server architecture using passive or minimally intrusive techniques.