Original Research · AI Search

We Checked 56 of America’s Biggest Brands: Newsrooms Lock AI Out, Everyone Else Leaves the Door Open

By Ravinder Cheema, Clicks Dynasty · October 8, 2026 · Data collected October 8, 2026

Every website has a small public file called robots.txt that tells bots what they may read. In 2026 that file decides something big: whether ChatGPT, Claude, Perplexity and Google's AI can learn from a brand, quote it, and recommend it to customers.

So we checked. On October 8, 2026 we pulled the robots.txt of 56 of America's best-known brands, eight in each of seven industries, and logged every rule aimed at 13 named AI crawlers. Here's what we found.

Finding 1: Most big brands haven't made an AI decision at all

Two out of three brands (38 of 56) don't name a single AI crawler anywhere in their robots.txt. Their general rules still apply to AI bots, so in practice they're open, but nobody wrote a rule on purpose.

• Banking & payments: 8 of 8 silent. Chase, Bank of America, Wells Fargo, Citi, Capital One, American Express, PayPal and Charles Schwab have no AI-specific rules.
• Restaurants: 8 of 8 silent. McDonald's, Starbucks, Chipotle, Domino's, Subway, Chick-fil-A and Wendy's have none. Taco Bell's robots.txt returned “not found”, so it has no rules for any bot.
• Airlines and hotels: 6 of 6 silent. Delta, United, American, Southwest, Marriott and Hilton.

Finding 2: Newsrooms are the ones locking the door, at 12× the rate

Six of the eight news and media sites we checked fully block at least one AI crawler or refused our AI reader outright: The Washington Post, Forbes, ESPN, NPR, CNN and NBC News. Only USA Today had no AI rules. CNBC's server returned an error, so we counted it as neither.

Outside news, just 3 of 48 brands do the same: Airbnb, WebMD and Amazon, whose robots.txt refused our AI reader. That's 75% versus 6%. Publishers sell content, so they guard it. Everyone else sells products and services, and so far they mostly let AI in.

Finding 3: Nobody blocks AI search, and the smartest brands split the difference

Not all AI crawlers do the same job. Training crawlers (GPTBot, ClaudeBot, CCBot) collect pages to build AI models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so an AI can answer a question right now and cite the source.

Across the 52 files we could read, the most-blocked crawlers were all training bots. Not one fully blocked OpenAI's OAI-SearchBot or Anthropic's Claude-SearchBot by name.

Two brands show the strategy we'd call the smart split:

• Forbes fully blocks GPTBot, ClaudeBot and seven other crawlers, then explicitly allows OAI-SearchBot and Claude-SearchBot. Translation: “Don't train on us, but do cite us.”
• Airbnb fully blocks GPTBot, ClaudeBot and Applebot-Extended, while ChatGPT's search and assistant bots and PerplexityBot only face a few path limits.

Finding 4: A few brands roll out the welcome mat

• GEICO names 12 AI crawlers and gives each one Allow: /. It's the clearest “please read us” signal in the study.
• Costco names ClaudeBot only to slow it down with a crawl delay, not to block it.
• CVS keeps six AI crawlers out of just one section: its drug information pages. Everything else stays open.
• Expedia writes rules specifically for AI search and assistant bots, with path limits only.

Finding 5: robots.txt isn't the only gate

Amazon, CNN and NBC News refused our AI-assisted reader because of their robots.txt rules, and CNBC's server returned a “403 forbidden” error before we could read anything. That's a reminder that firewalls and CDNs can block AI bots even when robots.txt says nothing. If you want AI to find you, check both.

The full grid: 56 brands × 13 AI crawlers

Filter by industry, or open the full-size image to share it.

Full block Some pages blocked Explicitly allowed Not mentioned Refused our AI reader / error

What this means for your business

If the biggest brands in America haven't made a deliberate AI decision, most small businesses haven't either. Our Bakersfield AI crawler study found 1 in 14 local business websites turning away AI crawlers, often by accident through a security plugin or hosting default. That's in the same range as the 6% of big non-media brands here, using a different method.

For a local business, being invisible to AI search usually costs more than it protects. When someone asks ChatGPT or Perplexity for “the best plumber near me”, the AI can only recommend businesses it can read. Our California AI Visibility Index shows how different those recommendations already are between AI tools.

A sensible starting point

Allow AI search and assistant bots, and decide separately about training bots. Here's the “smart split” as robots.txt rules:

# Let AI search and assistants read and recommend you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

# Optional: opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Keep your existing rules for Googlebot and Bingbot, then check that your firewall or CDN isn't blocking these bots anyway. Our free SEO & AI checker tests that for you.

Methodology

• Sample: 56 well-known US consumer brands, 8 in each of 7 industries, chosen for name recognition. It is not a random or statistically representative sample.
• Date: every file was retrieved on October 8, 2026. Robots.txt files change often; this is a snapshot.
• What we checked: rules naming 13 crawlers: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, anthropic-ai (Anthropic); PerplexityBot; Google-Extended; Applebot-Extended; CCBot (Common Crawl); Bytespider (ByteDance); Meta-ExternalAgent; Amazonbot.
• Definitions: “Full block” means the crawler is named in a group containing Disallow: /. “Some pages blocked” means path-level rules only. “Not mentioned” means the crawler isn't named, so the site's general rules apply.
• How: we retrieved each file with an AI-assisted web reader and logged the rules for each crawler. Where a site's robots.txt refused that reader, we recorded it as “refused”. Three outlets we planned to include could not be retrieved by our tools and were replaced with other national outlets.
• Limits: robots.txt is a public request, not an enforcement layer; we did not test firewalls except where a server refused us. If you spot an error, tell us and we'll correct it.

Download the full dataset (CSV)

Free to reuse under CC BY 4.0. Cite as: Clicks Dynasty, “Big Brand AI Crawler Study,” October 2026, clicksdynasty.com/big-brand-ai-crawler-study.html

Frequently asked questions

What is an AI crawler?

A bot that reads websites for an AI company. Some collect training data (GPTBot, ClaudeBot, CCBot), some power AI search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and some fetch a page when a person asks an assistant to (ChatGPT-User).

Does blocking AI crawlers in robots.txt actually work?

For the major AI companies, mostly yes: OpenAI, Anthropic, Google, Apple and Perplexity say their named crawlers follow robots.txt. It is a request, not a lock, and firewalls or CDNs can block bots separately, as CNBC's server did in our test.

Should my business block AI crawlers?

Most local and service businesses should not block AI search crawlers, because those bots decide whether ChatGPT, Claude and Perplexity can read and recommend you. Blocking training-only crawlers is a separate, reasonable choice for publishers and anyone protecting original content.

Which AI crawlers did the brands in this study block most?

Training crawlers: GPTBot, ClaudeBot and Common Crawl's CCBot were each fully blocked by 5 of the 52 readable files. None fully blocked OpenAI's OAI-SearchBot or Anthropic's Claude-SearchBot by name.

Can I use this data?

Yes. The full dataset is free to download and reuse under CC BY 4.0. Please credit Clicks Dynasty and link to this page.

Want to know whether AI tools can find and recommend your business? See our AI SEO services or get in touch.

Brand names are trademarks of their respective owners. Clicks Dynasty is independent and not affiliated with any brand or AI company named here. This study reports what public robots.txt files said on one date; it does not describe any company's intentions.