We checked 100 Bakersfield business websites. 1 in 14 turn away AI crawlers.
ChatGPT, Perplexity and Claude read business websites when people ask them for a recommendation. We checked whether 100 independent Bakersfield businesses let those tools in, and we list every result below. Most do. But 7 turn at least one AI crawler away, and 10 more have technical problems that can get in the way.
Can AI tools read these websites?
Each step removes the sites that failed it. 87 of 100 websites passed every step.
Which AI tools get turned away
The 7 sites that turned away an AI crawler, by who was blocked.
robots.txt by the numbers
We could read 81 of the 100 robots.txt files. Percentages below are out of those 81, except where noted.
What we found
ChatGPT-User, which ChatGPT uses to open a page when someone asks about that business, as well as OpenAI's training crawler.ai-input=yes) or point AI tools to an llms.txt file. Everyone else allows AI only because their robots.txt doesn't mention it.Results by industry
| Industry | Sites | Turned away AI | Other problems | Welcomes AI | No problems |
|---|---|---|---|---|---|
| Veterinarian | 7 | 1 | 3 | – | 43% |
| Pest control | 4 | – | 2 | – | 50% |
| Restaurant | 2 | – | 1 | – | 50% |
| Accountant | 8 | 3 | – | – | 62% |
| Chiropractor | 6 | – | 1 | – | 67% |
| Electrician | 5 | 1 | – | 1 | 80% |
| Auto repair | 6 | – | 1 | 1 | 83% |
| Dentist | 6 | 1 | – | 1 | 83% |
| Orthodontist | 6 | – | 1 | 1 | 83% |
| Roofing | 6 | – | 1 | 1 | 83% |
| Pool | 7 | – | – | 1 | 86% |
| Lawyer | 9 | 1 | – | – | 89% |
| HVAC | 8 | – | – | 1 | 100% |
| Plumber | 6 | – | – | – | 100% |
| Landscaping | 4 | – | – | 1 | 100% |
| Med spa | 4 | – | – | 1 | 100% |
| Solar | 3 | – | – | 1 | 100% |
| Insurance | 2 | – | – | – | 100% |
| Real estate | 1 | – | – | – | 100% |
Industries with only 1 or 2 sites are too small to compare; they are shown for completeness.
Results by website platform
Identified from each site's robots.txt where possible. Sites we couldn't read are counted as "Other / not identified."
| Platform | Sites | Turned away AI | Other problems | No problems |
|---|---|---|---|---|
| WordPress | 40 | 2 | – | 95% |
| Cloudflare-managed | 4 | – | – | 100% |
| Wix | 4 | – | – | 100% |
| Squarespace | 3 | – | – | 100% |
| Joomla | 1 | – | – | 100% |
| Drupal | 1 | – | – | 100% |
| Other / not identified | 47 | 5 | 10 | 64% |
All 100 websites
Is your business on this list? We'll check and fix AI crawler access for free for any business in this study. Book a free check. Think we got your result wrong? Let us know and we'll recheck and update it.
| # | Industry | Result | What we found | Website |
|---|
How to check and fix your own site
Open yoursite.com/robots.txt in your browser. If you see Disallow: / under any of these names, that AI tool can't read your site: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Bingbot or Applebot. To let them in, add this to your robots.txt:
Blocking training crawlers such as GPTBot, ClaudeBot or CCBot is a separate choice. It keeps your content out of future AI models, and it doesn't stop AI search tools from recommending you.
Check your site in 10 seconds
Paste your robots.txt into our free checker and see which AI crawlers can reach you. Nothing is uploaded.
How we did this study
robots.txt file on September 28, 2026, with a second check on September 29, using an AI web-reading tool (Anthropic's Claude). We applied the same rules our open-source tool local-seo-lint uses.Data license: free to reuse under CC BY 4.0. Please credit "Clicks Dynasty, Bakersfield AI Crawler Study" and link to this page. Related: California AI Visibility Index.
Is AI recommending your business?
Being readable is step one. Book a free 20-minute AI Visibility Check and see what ChatGPT and Perplexity say when customers ask for a business like yours.