Basic errors across websites are driving billions of unnecessary machine requests. Removing 3 in 4 of them saves millions in AI spend and tonnes of CO2 a year, the same as taking 619,000 cars off the road in the US for a day. It also reduces misinformation risk, as current information becomes easier for machines to identify. Globally this could mean saving over 40 trillion tokens.
01Executive summary.
AI leaders are being asked to scale capability while controlling cost. Most efficiency programmes look at models, tokens, GPUs and data centres. The research points to another place: an organisation's own information estate can make unchanged material look changed, creating repeat machine work before a model produces anything useful.
Across the 209 million US business, government and education websites in the model, the research estimate of more than 1 in 3 has been modelled at exactly 1 in 3. That produces a baseline of 69.7 million sites. If repeat retrieval happened once a week, the headline opportunity becomes:
The result: 3.62 billion repeat machine retrievals a year. Removing 3 in 4 of them saves 13.59 trillion tokens (US) and 7,139 tonnes of CO2, the same as taking 619,000 cars off the road for a day.
The assumptions and supporting calculations are set out at the end. The question for leaders is whether existing AI and digital infrastructure is being spent rediscovering information that has not changed.
When we stop rushing to embrace AI, and address some of the basics that are creating waste and misinformation, we save millions, reduce CO2 and make a dent in the misinformation problem.Lawrence Shaw, Chief Vision Officer, AAAnow
02The waste starts before the model.
Machines need reliable ways to distinguish genuine change from information they have already seen. Internet infrastructure has long supported mechanisms that let a machine check whether something has changed without retrieving the complete item again. When the signals describing change are unreliable, some of that efficiency can disappear.
For a chief executive or Head of AI, the mechanism is secondary. The business consequence is that an organisation can invest heavily in AI efficiency while its own information estate creates work that adds no new information. That work can begin with another request and another transfer, then continue into search, indexing or AI processing where the receiving system chooses to do more.
Further savings come from the second source of waste, which sits alongside false freshness: failed retrievals. Up to 28% of machine requests fail through broken links, 404 responses and related access errors. For AI systems, that creates 2 problems. The missing information weakens trust in the organisation's digital estate, while retries, alternative searches and additional processing consume resources without resolving the original request. On the same model, reducing this failure rate removes a further 4% of AI spend and up to 6% of associated CO2.
The finding is precise: inaccurate freshness signals defeat established efficiency mechanisms and make unchanged information more expensive for machines to recognise. The research stops short of claiming that a changed date on its own triggers a complete AI processing cycle.
03Scale turns a small defect into a management issue.
One unnecessary retrieval against 1 page barely matters. Repeated across tens of millions of organisational websites, the same behaviour becomes a different management problem. The US model reaches 3.62 billion unnecessary retrievals each year at 1 repeat retrieval per site each week.
At an illustrative 1 MB response, that weekly model represents 3.62 petabytes of transferred data. Removing half would avoid 1.81 billion retrievals and 1.81 petabytes before any downstream AI-processing benefit is counted. The reduction scenarios are shown at the end so the value can be judged with the assumptions in view.
Useful crawling, search and AI activity stay as they are. The objective is to make genuine change easy to recognise and unchanged information inexpensive to dismiss, so the organisation keeps the value of AI while removing repetitions that contribute nothing new.
04AI efficiency is becoming a board-level cost question.
The International Energy Agency projects global data-centre electricity use at 945 TWh by 2030, with accelerated servers driven mainly by AI responsible for almost half of the projected net increase. The United States is expected to see data centres account for almost half of electricity-demand growth to 2030.
Investment is moving at the same pace. The IEA reported that capital expenditure by 5 large technology companies exceeded $400 billion in 2025 and is expected to rise sharply again during 2026. Against that backdrop, an information estate that encourages avoidable machine work becomes an efficiency issue worth measuring.
The financial opportunity can become material even when only a fraction of repeat retrievals reaches paid AI processing. Using 5,000 input tokens as the downstream assumption and current commercial input prices between $1 and $4 per million tokens, a 50% reduction in the weekly US scenario produces an illustrative annual saving between $9.1 million and $36.2 million. That excludes network, server, storage, indexing and output-token costs.
05The improvement opportunity is measurable.
A Head of AI can turn the issue into 4 practical questions.
- Where is unchanged content being presented as changed?
- How often are machines returning to unchanged material?
- How much of that repeat activity can be removed without reducing useful discovery?
- What downstream cost disappears when the repeat work stops?
Answering those questions means measuring repeat bot requests, the proportion of unchanged resources returned in full, the stability of freshness signals, and the extent to which repeat retrieval flows into search, indexing or AI processing. The cause can sit within publishing, application, metadata, caching or delivery infrastructure, so diagnosis comes before remediation.
Once the cause is known, this ranks among the quickest wins available for carbon and AI cost. For the majority of sites the fix is a server setting or a template change for pages, done without touching the model, the hardware or the data centre.
The gain is fewer pointless repetitions of unchanged information, with useful AI interactions untouched. For organisations already asked to justify AI operating cost, infrastructure demand and environmental impact, that is a direct efficiency question.
06Measure the waste before buying more capacity.
AI efficiency programmes focus on model choice, token usage, hardware utilisation and data-centre design. Those levers matter. They begin too late when the information feeding the system is already creating avoidable work.
Before approving more capacity, more inference spend or another optimisation programme, leaders should ask how much existing capacity is being spent rediscovering information that the organisation already knew had not changed. That number is measurable, and so is the saving.
07Evidence, assumptions and calculations.
The figures below set out the modelling assumptions. They are scenarios built from the research and the US example set of 209 million sites. They show modelled exposure, presented as scenarios, with no claim that machines respond identically to freshness information.
1. Working baseline
| Item | Working value |
|---|---|
| US example set | 209 million business, government and education websites, excluding personal and smaller SME sites. |
| Research prevalence estimate | More than 1 in 3 examined environments exhibit the identified behaviour. |
| Modelling prevalence used | Exactly 1 in 3, held below the research estimate. |
| Sites in US modelling baseline | 69.7 million websites. |
| Global context | The wider web is put at 1.8 billion websites. |
2. Repeat retrieval scenario
| Illustrative frequency | Additional retrievals each year | Transfer at 1 MB each |
|---|---|---|
| One additional retrieval each month | 836 million | 0.836 PB |
| One additional retrieval each week | 3.62 billion | 3.62 PB |
| One additional retrieval each day | 25.43 billion | 25.43 PB |
The 1 MB response size is an illustrative full-response assumption. At 5 MB per response, the weekly scenario represents 18.11 PB of annual transfer.
3. What different reduction levels could remove (US)
Globally the figures would be at least 3 times these, measured against the global websites in proportion to those that are business, government or education.
| Reduction scenario | Retrievals avoided | Transfer avoided at 1 MB | Input tokens avoided at 5,000 each |
|---|---|---|---|
| 25% reduction | 0.906 billion | 0.906 PB | 4.53 trillion |
| 50% reduction | 1.811 billion | 1.811 PB | 9.06 trillion |
| 75% reduction | 2.717 billion | 2.717 PB | 13.59 trillion |
Read as input tokens avoided each year in the US weekly scenario: 4.53 trillion at a 25% reduction, 9.06 trillion at 50%, and 13.59 trillion at 75%.
4. Illustrative downstream AI-processing value
The cost model assumes that each avoided retrieval would otherwise have contributed 5,000 input tokens to a downstream paid AI process. That figure is a modelling assumption and makes no claim about crawler behaviour. Current commercial list prices provide the cost range used here: OpenAI GPT-6 Sol is listed at $1 per million input tokens, Anthropic Claude Sonnet 5 at $2, and OpenAI GPT-5.6 Sol at $4.
| Reduction scenario | $1 per million input tokens | $2 per million | $4 per million |
|---|---|---|---|
| 25% reduction | $4.5m | $9.1m | $18.1m |
| 50% reduction | $9.1m | $18.1m | $36.2m |
| 75% reduction | $13.6m | $27.2m | $54.3m |
These figures are processing-cost proxies, modelled against list prices. They exclude data transfer, origin infrastructure, CDN services, storage, indexing, retrieval orchestration and any output-token cost.
5. Electricity and CO2 scenarios
A 2026 Joule study estimates median energy use of 0.31 Wh for a standard frontier-scale inference query and 3.91 Wh for a long test-time-scaling query. The calculation below treats 1 avoided retrieval as 1 query-equivalent, to show the scale where redundant retrieval propagates into downstream inference. The research stops short of establishing that each retrieval does so.
| Reduction scenario | Standard-query energy | CO2 avoided | Long-reasoning energy | CO2 avoided (long) |
|---|---|---|---|---|
| 25% reduction | 281 MWh | 189 tonnes | 3.54 GWh | 2,380 tonnes |
| 50% reduction | 562 MWh | 377 tonnes | 7.08 GWh | 4,759 tonnes |
| 75% reduction | 842 MWh | 566 tonnes | 10.62 GWh | 7,139 tonnes |
CO2 uses the US EPA national marginal electricity factor of 6.72 × 10-4 metric tonnes of CO2 per kWh for avoided or reduced electricity consumption. The marginal factor is used because the scenario concerns electricity that would not need to be consumed. The car equivalent uses the EPA figure of 4.21 metric tonnes of CO2 per passenger vehicle each year: 7,139 tonnes is the same as taking 619,000 cars off the road for a day.
6. Evidence boundaries
The research supports the proposition that reliable modification information and validators reduce unnecessary transfer, and that inaccurate freshness signals defeat date-based revalidation in systems that rely on those signals. Google also documents the resource benefit of 304 Not Modified responses, while OpenAI documents an explicit desire to avoid duplicative crawling.
Public documentation from OpenAI, Anthropic, Perplexity and other AI providers stops short of stating that a changed Last-Modified value automatically triggers a recrawl, re-index, re-embed or inference call. The AI-processing, electricity and CO2 figures above are modelled from documented mechanisms.
Source and link register.
Sources used for the additional cost and emissions model
- International Energy Agency, Energy and AI.2025. Energy demand from AI.https://www.iea.org/reports/energy-and-ai/energy-demand-from-aiiea.orgWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- International Energy Agency, Key Questions on Energy and AI.2026.https://www.iea.org/reports/key-questions-on-energy-and-aiiea.orgWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- US Environmental Protection Agency, Greenhouse Gas Equivalencies Calculator methodology.Calculations and references.https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator-calculations-and-referencesepa.govWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- Oviedo et al., Energy use of AI inference, efficiency pathways, and test-time scaling.Joule, 2026.https://www.sciencedirect.com/science/article/pii/S2542435126001145sciencedirect.comWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- OpenAI API pricing, GPT-6 Sol and GPT-5.6 Sol.Current list prices.https://developers.openai.com/api/docs/pricingdevelopers.openai.comWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- Anthropic, Claude Sonnet 5 pricing.Current list price.https://www.anthropic.com/news/claude-sonnet-5anthropic.comWe do not enable external links - please copy and paste, you are then certain as to the link being opened