Cloudflare AI Bot Defaults Changed Sept. 15: Don’t Block Google by Mistake

On September 15, Cloudflare’s new AI crawler settings took effect. If your website runs through Cloudflare, take fifteen minutes this week to confirm the new Cloudflare AI bot defaults are not quietly blocking the search crawlers that bring you customers.

Most local service businesses never log in to Cloudflare. A web developer or hosting company set it up years ago, and it just works. That is exactly why a default change deserves a quick check.

What changed on September 15?

Cloudflare announced the change on July 1. Under the new default for newly onboarded domains, search crawlers are allowed, while AI training bots and AI agent bots are blocked on pages that show ads. Sites could opt out before September 15, when the new defaults took effect.

There is a wrinkle about who is affected. Cloudflare’s own blog describes the default in terms of new domains. Engadget reported that free accounts also switch to the new defaults. Those two accounts don’t line up neatly, so do not assume your site is exempt just because it has been on Cloudflare for years. Check the settings either way.

Cloudflare is also reframing its Pay Per Crawl program. According to PPC Land, the idea is moving toward paying when content is used in answers, with Ceramic.ai and You.com named as partners. That matters more to publishers than to a roofer or an electrician, but it shows where the crawler economy is heading.

Can Cloudflare AI bot settings block Googlebot?

Yes, in some setups. This is the part local businesses need to understand. Some companies run “mixed-use” crawlers that collect pages for search and for AI at the same time. According to Cloudflare, mixed-use crawlers are treated by their most restrictive behavior.

In plain English: if a site blocks AI training, Googlebot, Applebot and BingBot can end up blocked too. Those are the crawlers that feed Google, Apple and Microsoft search. For a business that depends on people finding it through search, blocking them by accident is far more costly than anything an AI training bot could take.

Notice what the default actually targets: training and agent bots on pages that show ads. Many plumbing, HVAC or cleaning websites don’t run display ads, so that part may never touch you. The risk is a training block that you, a past developer or a security plugin turned on, and that now catches mixed-use search crawlers along with it.

Training, search and AI answers: three different switches

A lot of confusion comes from treating “AI” as one setting. It isn’t. Here are the main controls in play right now and what each one governs.

The Search Console control went live for every site on August 31. Include is the default, and excluding a property removes its traffic and impressions from those AI features, typically within one to two days. It does not govern AI training; Google handles that separately through Google-Extended. Google was explicit that the setting doesn’t hurt you elsewhere:

“this control isn’t used as a ranking or inclusion signal affecting other parts of Search.”

Google, Search Console Help

We covered whether a local business should ever use that switch in our post on opting out of AI Overviews. Our answer was almost never, and the same logic applies here: if you want to limit AI training, use a training control, not a block that can catch search crawlers.

Does llms.txt or a robots.txt signal fix this?

Not really, at least not yet. llms.txt files are spreading fast. Originality.ai counted 36,120 sites with one in May 2026, up from 4,088 in June 2025, an 8.8x jump, according to PPC Land’s write-up. But Ahrefs’ study of 137,000 domains found that 97% of llms.txt files got zero requests in May 2026. Google has also noted that llms.txt is not required for Search.

Cloudflare’s own Content Signals directive for robots.txt has a similar problem. Search Engine Roundtable reported in July that Google said the directive “has no effect.” A signal only works if the crawler you care about reads it.

A more formal option is in progress. On September 14, the IETF published draft-08 of its AI Preferences vocabulary. It defines three categories, train-ai, ai-use and search, each set to y or n, and the search category overrides the other two. How those preferences attach to robots.txt is covered in a separate draft. The document itself says it does “NOT REFLECT CONSENSUS,” and it expires on March 18, 2027. It is worth watching, not worth building on today.

How to check your Cloudflare settings in 15 minutes

You don’t need to be technical for this. You need dashboard access, or ten minutes of your web person’s time.

1. Confirm whether you use Cloudflare. Ask your developer or host, or check who manages your domain’s DNS. If the answer is Cloudflare, keep going.
2. Find the AI crawler and bot settings. Look for the controls covering AI training bots, AI agents and search crawlers. Confirm that search crawlers are allowed.
3. Look for an AI training block. If one is on, find out who turned it on and why. Because mixed-use crawlers are treated by their most restrictive behavior, a training block can catch Googlebot, BingBot and Applebot.
4. Review your robots.txt file. Make sure nothing in it disallows the major search crawlers from your service and location pages.
5. Watch Search Console for a few weeks. A sudden drop in crawling or indexed pages after mid-September is a sign to recheck the Cloudflare settings first.
6. Write down your decision. Note who checked, what was set and why. The next time a default changes, you will know your starting point.

Should a local business block AI bots at all?

For most local service companies, no. Your website is marketing. Your service pages, prices, service areas and reviews are meant to be read, quoted and recommended. Being cited by AI answers is part of how customers now find contractors, which is why we focus on answer engine optimization rather than on locking pages down.

There are exceptions. If you publish original research, paid courses or content you license, you may have good reasons to limit AI training. Even then, use the narrowest control that does the job, and never one that can block search crawlers. If you are unsure how the change has affected your traffic, our analytics and tracking service can separate crawler problems from normal seasonal swings. For how Google treats sites generally, see our recap of the August 2026 spam update.

A default change is easy to miss and easy to fix. Fifteen minutes now beats discovering a crawl problem in November. If you would like us to look at your setup, contact MarketPros.AI.

Frequently asked questions

Related articles