Blocking Google would be treated as a commercial emergency by any business: visibility, traffic and revenue would fall together. Most organisations invest in the opposite, making themselves easier to find.
Many of those same organisations are turning AI away from their own digital estate. The information they have established as current and authoritative sits out of reach of the AI systems their customers, investors and candidates now consult, and AI builds its account of them from other material. This article draws on Lawrence Shaw’s piece for CMSWire, published 24 Sep. 2026.
I observed one organization commit £350,000 towards improving AI visibility so that it was being found and referenced in AI responses. However, its digital estate was not able to be fully accessed by the very AI tools where it was working to extend its reach.Lawrence Shaw, CMSWire, 24 Sep. 2026.
01A gap the organisation creates itself.
Much of what AI gets wrong about an organisation starts on the organisation’s own estate: content that is out of date or contradictory, PDF documents that machines misread, and routes AI cannot use. AAAnow’s research finds that 79% of misinformation or misrepresentation has the organisation’s own online position as its root cause. AAAnow Research, Dec. 2023 to Jan. 2026; P&C (human misinformation) 2014 to 2023.
Blocked access sits alongside these causes, and it is widespread. Automated traffic (bots, scrapers and crawlers) made up an estimated 53% of all internet traffic in 2025, and part of that traffic is the AI crawlers gathering the information that shapes AI answers. Research from US Tech Automations found that 19.1% of the 100,000 top-ranked websites explicitly block at least 1 of the top 21 AI crawlers in their robots.txt file, with OpenAI’s crawlers blocked most often.
02Turned away by decision, and by accident.
Some blocks are decisions: to keep content out of model training, to protect paid or licensed material, to manage server load, or to keep personal data and secure areas away from automated agents. Others are accidents, and these are the ones that go unseen.
Where accidental blocks come from
- Bot and scraper rules set up years ago, now catching AI crawlers.
- CDN and security settings that do their job a little too efficiently.
- JavaScript frameworks that AI crawlers often cannot render.
- Errors in robots.txt, and site-wide meta tags nobody knows are there.
- IP or location blocking, third-party plug-ins and rate limits that drop legitimate AI requests.
- Authenticated areas, paywalls and CAPTCHA.
Each has a sound origin. Together they decide what AI can read of the organisation, usually without anyone having decided it.
03Healthy to people, closed to AI.
The testing behind the CMSWire piece used 24 agents to assess an organisation’s visibility to 15 key AI tools, asking whether AI could reach the information the organisation has established as current and authoritative. Human access was tested alongside. Four patterns emerged.
1 · Selective restriction behind a healthy website
All human routes worked, while a number of AI-related routes were closed.
2 · Widespread refusal across AI-related routes
Human access was fine; the majority of AI tools were turned away.
3 · No reply
Of 14 AI-related routes, all but 1 received no reply, and 1 human route was also blocked.
4 · Google and Microsoft kept, the rest refused
Routes for Google and Microsoft stayed open; the other AI routes were refused.
Standard monitoring watches the human routes. A website can therefore report itself healthy while most of its AI routes are closed, and the gap stays out of sight until someone tests for it.
04What AI reads instead.
A blocked route leaves AI to answer from elsewhere. It draws on media coverage, historic pages left unblocked, and archived documents republished on other sites. That material may be superseded, and it rarely carries the messaging the organisation intends, which leads to both misinformation and misrepresentation.
With most of what AI says about a brand already drawn from other people’s pages, blocking the owned estate widens the gap at the single point where the organisation holds the authoritative account.
05Putting access right.
Access for AI carries commercial consequences, so it needs an owner, a policy and regular testing. The CMSWire piece sets out 5 steps.
1 · Test which AI routes are blocked, and why
Establish which AI tools are effectively blocked, separating deliberate decisions from settings nobody intended. Digital Doorway examines the relevant AI access routes with 24 agents, from the outside in.
2 · Correct the accidental blocks
Put right any route blocked inadvertently, and prevent it happening again.
3 · Revisit deliberate policies with their owners
Find out the reason behind each deliberate block and, where possible, reverse it. As the piece puts it, “the goalposts have moved and what may have made sense a few months ago now has consequences as more people use AI tools to search for information.”
4 · Assign the policy a clear owner
Blocking AI tools has a commercial impact, so ownership sits with a stakeholder accountable for visibility and revenue.
5 · Keep testing
AI keeps moving and website settings change. Continued testing shows emerging blocks before they shape what AI says.
06Where this sits for leadership.
What AI says about an organisation can only reflect what the organisation has approved if AI can reach it. Knowing which routes are open, which are closed, and who decided, closes part of the gap between what the organisation understands of itself and what AI sees of it.
That understanding turns an invisible setting into a decision the board can see and own. For organisations investing in AI visibility, it is also the check that the investment can reach the systems it is meant for. AI Accelerator establishes these foundations and shows where the organisation stands.
If this is an issue in your organization, then make it a priority to fix it.Lawrence Shaw, CMSWire, 24 Sep. 2026.
Sources.
The CMSWire piece this article is drawn from, and the research it cites. Addresses are shown for reference.
- CMSWire · Lawrence Shaw · Is Your Website Quietly Shutting Out ChatGPT and Claude Crawlers?24 Sep. 2026 · Editorial, Digital Experience. The source article: causes of AI blocking, testing patterns across 24 agents and 15 AI tools, and 5 steps to find and fix blocks. https://www.cmswire.com/digital-experience/is-your-website-quietly-shutting-out-chatgpt-and-claude-crawlers/CMSWireWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- US Tech Automations · Closing web indexJul. 2026 · 100,000 top-ranked websites tested against the top 21 AI crawlers in robots.txt; 19.1% block at least 1, OpenAI’s crawlers most often. https://ustechautomations.com/resources/blog/closing-web-index-july-2026US Tech AutomationsWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- Imperva · 2026 Bad Bot Report2026 · Automated traffic estimated at 53% of all internet traffic in 2025. https://www.imperva.com/resources/resource-library/reports/2026-bad-bot-report/ImpervaWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- AirOps · LLM brand citation trackingThird-party pages, rather than brand-owned domains, were the source of 85% of brand mentions in AI responses. https://www.airops.com/blog/llm-brand-citation-trackingAirOpsWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- AAAnow Research · Root cause of misinformation and misrepresentationThe organisation’s own online position is the root cause of 79% of misinformation or misrepresentation. AAAnow Research, Dec. 2023 to Jan. 2026; P&C (human misinformation) 2014 to 2023.
- AAAnow · Digital DoorwayExamines relevant AI access routes with 24 agents and establishes whether important organisational information can be reached through them. https://aaanow.ai/platform/digital-doorway/ (opens in a new tab)