Is your website blocking ChatGPT by mistake?

Like Google, AI companies send programs called crawlers to read websites. Each crawler has a name, and a file on your site called robots.txt tells each one whether it may come in.

OpenAI and Anthropic each run three. One finds pages to show in search answers, one gathers pages to train their models, and one fetches a page when someone asks about it.

Google’s AI Overviews come from Google Search itself, so they use Googlebot, the same crawler behind your rankings.

For a business that wants to be recommended, the search crawlers matter most. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce a site’s visibility in Claude’s results.

The training crawlers are a separate choice. You can block GPTBot and ClaudeBot to keep your pages out of model training and still let the search crawlers in.

1. Open yourbusiness.com/robots.txt in a browser, with your own address in place of yourbusiness.com.
2. Look for a User-agent line naming OAI-SearchBot or Claude-SearchBot, or an asterisk, which means every crawler.
3. Under it, Disallow: / blocks the whole site. Allow: /, or Disallow: with nothing after it, lets the crawler in.

If your site has no robots.txt at all, every crawler is allowed. After a change, OpenAI says it can take about a day for ChatGPT to notice.

Security services can block AI crawlers before they reach your site. Cloudflare began blocking them by default on new sites in July 2025, so if your site runs through Cloudflare, check its AI crawler settings as well as your robots.txt. Website builders like Squarespace have a similar switch.

Letting the crawlers in is the first step to being recommended. The rest is on the AI recommendations page.