Your firewall may be blocking AI crawlers
Your website has two doors in front of it. The first is robots.txt, the plain text file that says which software may read your pages. The second is a security service that sits in front of the whole site and decides what is allowed to reach it. You may have opened the first and never known the second exists. The second can still turn AI crawlers away.
The cost is the same as any other block. When someone asks an assistant to recommend a business like yours, you are not in the list.
A block you did not choose
Since July 2025, Cloudflare blocks known AI crawlers by default on newly added domains. If someone else set up your site and it runs through Cloudflare, that default may still be on.
Nothing looks wrong. Your site loads. Visitors are fine. Only the crawlers that feed AI search are turned back, and no one told you.
Why it is easy to miss
The door in front does a real job. It filters junk traffic and keeps the site online under attack, so nobody thinks to question it. Your developer added it for good reasons. The AI crawlers were caught in the same net.
Find who holds the switch
Ask three questions, in order.
- Who built the site? Your developer or agency. Ask whether a security service was placed in front of it, and who owns the account.
- Who hosts it? Your host may filter bots as part of the plan. Ask them to confirm whether AI crawlers are allowed through.
- Do you have a Cloudflare account? One may have been created under your business email when the site launched, and you have never logged in.
You do not need to read any settings. You need to know who controls the layer. If all three answers are “I don’t know”, start with the host. They can see what runs in front of the site.
What to ask for
Ask them to let the AI search crawlers through. Three matter most for being named in an answer: OAI-SearchBot, Claude-SearchBot and PerplexityBot. They read pages so an assistant can quote and cite them.
They are not the same as the training crawlers, which collect pages to build future models. Blocking training is a fair choice. Blocking search crawlers quietly removes you from answers. When they confirm, ask them to name which crawlers are allowed, not just that bots are fine.
A check you can run now
On your phone, ask ChatGPT or Perplexity for your service in your city. If the answer names other businesses and not yours, something is standing between you and the answer.
Then open yourdomain.com/robots.txt in a browser. If it loads and welcomes crawlers, the file is not the problem, and the layer in front is the place to look. There is a full guide to that file in robots.txt for AI crawlers.
The 10-minute check pulls all of this into one pass: is your site visible to AI?