Is your site blocking AI crawlers? A 5-minute check
October 10, 2026
Open a new browser tab and type yoursite.com/robots.txt (replace yoursite.com with your domain). What you see is a small text file that tells automated crawlers which parts of your site they may visit. Most site owners have never looked at theirs. Some would be surprised by what it says.
What to look for
Scan the file for the names of AI crawlers: GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended. If you see any of them next to Disallow: /, your site is telling that AI system to stay out entirely. Sometimes a broader rule like User-agent: * with Disallow: / blocks everything, AI included.
Should you allow them?
For most local and service businesses, yes. Blocking AI crawlers does not just hide you from chatbots. It hides you from the assistants your future customers ask for recommendations. There are legitimate reasons to block (publishers protecting paywalled content, for example), but "my security plugin did it by default" is not one of them.
How to fix it
On WordPress or a custom site, edit robots.txt to explicitly allow the AI crawlers you want, or at least remove the rules blocking them. On hosted platforms like Squarespace or Wix, the platform controls robots.txt, and most already allow AI crawlers by default, so check before worrying.
Our free scan checks your robots.txt posture automatically, along with six other agent-readiness checks. It takes less than the five minutes you just spent reading this.