“Are we blocking AI?” sounds like a technical yes-or-no question. For a marketing website, it usually contains several decisions: whether search services can discover public pages, whether content may be used for model training and whether private information is protected. The implementation should follow those decisions separately.
Can a website allow AI search while opting out of training?
For OpenAI, the documented settings are independent. Its crawler reference distinguishes OAI-SearchBot, associated with search, from GPTBot, associated with potential model-training use. A webmaster can allow the former while disallowing the latter.
That distinction should appear in the team's decision record. “Allow AI” is too broad to describe the intended policy. Record the service, the purpose, the affected paths and the owner. Other providers require their own current documentation rather than an assumed equivalent setting.
Our robots.txt permits the crawler. Why might discovery still fail?
The permission file is one part of the access path. A firewall might challenge the request. A redirect might lead to the wrong regional page. The server might return an error, or the delivered page might omit the product facts a reader expects to find.
Work through an actual priority URL. Inspect the applicable robots rule, the response, the redirect destination and the content delivered. Repeat for a product page, a comparison and a regional landing page where the site uses different delivery patterns. A successful request to the homepage does not verify the whole site.
OpenAI's publisher FAQ advises keeping OAI-SearchBot unblocked for inclusion in summaries and snippets. Access enables consideration. It does not establish that a particular page will be selected.
Does changing the user-agent prove that a bot can access the site?
It proves how the server handled that particular request with that header. It does not prove that the genuine crawler can reach the same URL from its own infrastructure. A firewall can treat those requests differently.
Use synthetic requests as diagnostics, then inspect real server logs where available. Verify bot identity using the provider's documented methods. Keep the evidence precise: “our test request returned the full page” is different from “a verified search crawler successfully fetched this URL.”
Should the same rule be applied to Google?
No provider's terminology should stand in for another provider's controls. Google maintains specific guidance for its AI Search features. Use the current instructions for the experience the business wants to influence. Search inclusion, previews and training-related controls need to be understood in their own documented context.
This is a good place for marketing and engineering to work together. Marketing explains the publishing objective. Engineering identifies which technical settings implement it and provides evidence that they work. Neither team should have to infer the business policy from a configuration copied into the site years ago.
Can robots.txt protect private content?
Private content should use the site's authentication and authorisation mechanisms. A crawler instruction does not replace access control. Public product information and a private distributor portal therefore require different handling even if both contain commercially important material.
When should content optimisation resume?
Once the important public pages have a verified access path, investigate relevance, product evidence and source quality. If the pages were already accessible, avoid turning crawler settings into an explanation for every visibility gap.
For Frontier's GEO audits, the technical finding should be reproducible: which URL failed, what happened, which setting was responsible and what changed after correction. That gives the content team a dependable foundation for the next stage of work.