Salesforce B2B Commerce natively provides a robots.

A customer can create its ownrobots.txt file for three valid reasons:

The customer wants to reference multiple storefront sitemap indexes in a single robots.txt file. This can be useful if the customer has multiple storefronts under the same domain and wants to provide a single entrypoint for search engines to crawl their sitemaps.

The customer wants to reference a custom sitemap index. This can be useful if the customer has created their own sitemap index that contains custom sitemaps or sitemaps from other sources.

The customer’s store is not located at the root of their domain. This can be useful if the customer has their store under a subdirectory or a subdomain and wants to specify a different robots.txt file for their store than for their main domain. Salesforce References: B2B Commerce and D2C Commerce Developer Guide, Robots.txt File