Akamai: serving pages as Markdown to AI agents

01What it does, and why you need it.

When a request arrives with the header Accept: text/markdown, the edge fetches your HTML, strips navigation, scripts and styling, and returns Markdown. An AI system reads the same content in a fraction of the tokens.

Akamai’s own data shows AI bot traffic rising fast and each agent request fanning out into many page fetches. Serving them Markdown shrinks what you send and what they read, and you choose who gets it.

02Who should deal with this.

Three people.

Property administrator

The Akamai property administrator, because the work sits in Property Manager and EdgeWorkers.

Security owner

The security owner of Bot Manager, because Akamai’s approach to AI traffic is built around deciding which bots get what, and the Markdown response is one of the things you can decide to serve.

Content licensing

Whoever owns content licensing, because you will be setting your own usage policy header rather than inheriting a default.

Level of importance

This is a development task, not a toggle, so budget days rather than minutes. The value case is strongest for content-heavy sites that already see meaningful AI fetcher and agent traffic in their Bot Manager reporting. If that traffic is small, publish static Markdown versions of key pages and revisit later.

03The Akamai position on AI traffic.

Akamai has published its own measurements, and they are the right starting point for deciding whether to invest.

May 2025

Akamai reported that its bot management team had been tracking AI web scraping since Nov. 2024. AI scraping accounted for 0.1% of daily traffic across the Akamai network, which on a network handling more than 1 billion requests a day meant more than 600 million requests handled by its application security protections. The first wave hit commerce customers, because inventories and prices change constantly and models revisit them, but by early 2025 growth in business services, gambling and healthcare was outpacing commerce. Akamai also noted the fan-out effect: a single question to an agent or a retrieval-augmented model can trigger many parallel requests, and cited researchers putting that at upwards of 100 scraping requests per user or agent request.

Jul. 2025

Akamai reported AI bot traffic had grown 300% in a year, reaching 0.27% of traffic across its platform and 0.9% of known bots, already in the billions of requests per day. Publishing customers were more than 7 times more likely to see AI-driven bot traffic than the broader customer base. Akamai’s framing in that post is that how you manage your relationship with AI and LLM bots is a business decision, not just a security one: which LLMs to serve, which use cases matter (model building versus users and agents), which information to share, and which mechanisms to use.

3 Sep. 2026

Akamai split the single AI Bots category in its Bot Directory into 3: AI training crawlers (for example GPTBot, ClaudeBot, Meta-ExternalAgent), AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and AI fetchers and agents (ChatGPT-User, Claude-User, Perplexity-User, ChatGPT Agent, GoogleAgent-Mariner). The stated purpose is to let customers apply different policies based on what a bot is there to do, for example restricting training crawlers while allowing search indexers and trusted agents.

This is where Akamai differs from Cloudflare in approach. Cloudflare ships a converter and a default permission header. Akamai ships classification and control, and leaves the conversion to you. The practical consequence is that on Akamai, Markdown for Agents is best treated as a policy outcome: this category of AI traffic gets Markdown, that category gets HTML, that category gets a challenge.

04How it works: content negotiation.

The mechanism is standard HTTP content negotiation, the same one a browser uses to say which image formats it accepts. A client that wants Markdown sends:

Request header

Accept: text/markdown

Coding agents including Claude Code, OpenCode and Gemini CLI already send this header when fetching pages, so a site that honours it serves Markdown to those tools with no change on their side. A browser sending a normal Accept header gets HTML as before.

Akamai’s own documentation site practises a simpler variant of the same idea. Each techdocs product publishes an llms.txt index, and any documentation page can be fetched as Markdown by appending .md to its URL.

05What the response should look like.

There is no Akamai-defined output format, so the sensible move is to match the shape Cloudflare documented, because AI systems are already being written against it:

1 · Frontmatter

YAML frontmatter with title, description and image taken from the page’s meta tags.

2 · Body

The body converted to Markdown, with headers, footers, navigation, scripts and styles stripped.

3 · Structured data

Any JSON-LD structured data preserved as a fenced json block at the end.

And these response headers:

Content-Type

text/markdown; charset=utf-8

Vary

Accept, so caches keep HTML and Markdown apart.

x-markdown-tokens, x-original-tokens

Estimated token counts that let an agent decide whether the page fits its context window.

content-signal

Your usage policy (see below).

Drop Content-Length, Content-Encoding, ETag and Last-Modified from the origin response, because they describe the HTML body, not the converted one.

06Content Signals.

Content Signals is a framework, at contentsignals.org, for stating how content may be used after it has been accessed, expressed as a content-signal header with 3 values: ai-train, search and ai-input. Cloudflare adds a default of all 3 set to yes when the origin is silent. On Akamai nothing is added for you, so decide the policy and set it, either at origin or with a Property Manager header behaviour. A publisher that has licensing arrangements with some AI vendors and not others will want this aligned with its Bot Manager policy rather than left blank.

Example header

Content-Signal: ai-train=no, search=yes, ai-input=yes

07How to build it on Akamai.

Route 1 · EdgeWorkers

Conversion at the edge.
This is the closest equivalent to Cloudflare’s feature. An EdgeWorkers function implementing responseProvider() takes the request, fetches the original HTML from origin as a sub-request, converts it and returns Markdown. Akamai documents this pattern under Response content transformation, including the caching pitfalls.

Property Manager rule:

1

Match on request header Accept containing text/markdown, and on path or content type as appropriate (HTML pages only; exclude APIs, assets and anything behind authentication).

2

Add the EdgeWorkers behaviour within that match.

3

Include a distinguishing header (Akamai’s docs use X-Subrequest: true) in the cache ID so the converted response and the original HTML are never stored under the same key. Bypass the EdgeWorkers behaviour when that header is present.

4

Add the Accept value to the cache ID for the converted response, or serve it uncached at first and add caching once tested.

Illustrative EdgeWorkers code. The conversion library is the part you choose; the pattern around it is Akamai’s documented one.

EdgeWorkers, responseProvider()

import { httpRequest } from 'http-request';
import { createResponse } from 'create-response';
import { convert } from 'markdown-for-agents';

const UNSAFE_RESPONSE_HEADERS = [
  'content-length', 'content-encoding', 'vary', 'etag',
  'last-modified', 'transfer-encoding', 'connection', 'keep-alive'
];

function wantsMarkdown(request) {
  const accept = request.getHeader('Accept');
  return accept && accept.join(',').toLowerCase().includes('text/markdown');
}

export async function responseProvider(request) {
  if (!wantsMarkdown(request)) {
    return httpRequest(request.url, { headers: { 'X-Subrequest': ['true'] } });
  }

  const origin = await httpRequest(request.url, {
    headers: { 'X-Subrequest': ['true'], 'Accept': ['text/html'] }
  });

  const contentType = (origin.getHeader('Content-Type') || [''])[0];
  if (origin.status !== 200 || !contentType.includes('text/html')) {
    return origin;
  }

  const html = await origin.text();
  const { markdown, tokenEstimate } = convert(html, { extract: true });

  const headers = {};
  for (const [name, value] of Object.entries(origin.getHeaders())) {
    if (!UNSAFE_RESPONSE_HEADERS.includes(name.toLowerCase())) headers[name] = value;
  }
  headers['Content-Type'] = ['text/markdown; charset=utf-8'];
  headers['Vary'] = ['Accept'];
  headers['x-markdown-tokens'] = [String(tokenEstimate.tokens)];
  headers['Content-Signal'] = ['ai-train=no, search=yes, ai-input=yes'];

  return createResponse(200, headers, markdown);
}

Points to check before relying on this:

Resource tiers

EdgeWorkers resource tiers set limits on CPU time, memory and response size. A full HTML-to-Markdown conversion of a large page may not fit the lower tiers. Test on real pages, not a hello-world.

Compression and Vary

Akamai’s docs warn that EdgeWorkers will not deliver gzip-compressed content to the CDN proxy, and that Vary headers can prevent caching in some configurations. Follow the Response content transformation guidance on which headers to strip.

The converter

Keep the converter conservative: preserve JSON-LD, strip everything else in script and style tags.

Logging

Log which bot categories are receiving Markdown, using DataStream 2, so you can see whether the traffic you are serving is fetchers and agents or training crawlers.

Route 2 · Static Markdown

At origin, delivered by Akamai.
If EdgeWorkers is not in your contract or the site is built from Markdown sources anyway, generate a .md file alongside each page at build time and let Akamai deliver it as an ordinary object. This is the approach Akamai uses for its own documentation and the one the llms.txt proposal recommends.

Then advertise it. Add a Link header with Property Manager’s Modify Outgoing Response Header behaviour on HTML pages. Optionally, add a Property Manager rule that redirects or rewrites a request carrying Accept: text/markdown to the .md object, which provides content negotiation without any code. Exclude the .md files from search indexing so they are not treated as duplicate content.

Link header, Route 2

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

Route 3 · Convert at origin

Cache at Akamai.
Middleware exists for common web frameworks that performs the conversion at origin when Accept: text/markdown is received. Akamai then caches the Markdown variant as long as the cache key includes the Accept header. This keeps the logic in your application code and uses Akamai purely for delivery and bot control.

08Tying it to Bot Manager.

Whichever route you take, the decision about who receives Markdown belongs in Bot Manager, not in the converter. With the 3 AI categories now in the Bot Directory, a workable starting policy is:

AI fetchers and agents

Allow, serve Markdown on request. These are acting for a person in real time.

AI search crawlers

Allow, serve Markdown on request, subject to your citation and licensing position.

AI training crawlers

Your call. Akamai’s example in its Jul. 2025 post is a retailer that serves product pages without pricing to one LLM and tarpits others; the same conditional actions apply here.

Impersonators

Akamai reports bots setting user agents to known LLMs to bypass detection, and detects them on additional signals. Do not gate the Markdown path on user agent alone.

09Testing.

Test

curl -s -D - https://www.example.com/some-page/ \
  -H "Accept: text/markdown" -o /dev/null

Check for Content-Type: text/markdown, Vary: Accept and your content-signal value. Then request the same URL with a browser Accept header and confirm HTML comes back and that neither variant has polluted the other’s cache.

10Sources.

01Akamai, May 2025. The Rise of the LLM AI Scrapers: What It Means for Bot Management. Akamai’s measurements of AI scraper traffic, growth by industry, the fan-out from agent requests and its recommended management approach. www.akamai.com/blog/security/rise-llm-ai-scrapers-bot-management
02Akamai, Jul. 2025. AI and LLM Bot Management Has Become a Business-Critical Issue. The 300% growth figure, publisher exposure, and the argument that managing AI bots is a business decision with worked examples for retailers and publishers. www.akamai.com/blog/security/ai-llm-bot-management-has-become-business-critical-issue
03Akamai, 3 Sep. 2026. The split of the Bot Directory AI category into training crawlers, search crawlers, and fetchers and agents, with the named bots in each. www.akamai.com/blog/security/introducing-more-granular-controls-ai-bot-traffic
04The Akamai Bot Directory of verified bots and their categories. techdocs.akamai.com/bot-manager/docs/verified-bots
05Akamai’s Bot & Agent Control solution page, covering Bot Manager, Content Protector, Account Protector and AI Brand Presence. www.akamai.com/solutions/bot-and-agent-control
06Bot Manager product page. www.akamai.com/products/bot-manager
07Akamai’s EdgeWorkers tutorial on response content transformation, including responseProvider(), in-place transformation, the caching pitfalls and the header handling used in the code above. techdocs.akamai.com/edgeworkers/docs/transform-response-content
08EdgeWorkers content transformation FAQ, including the html-rewriter module and streaming behaviour. techdocs.akamai.com/edgeworkers/docs/content-transformation
09EdgeWorkers product page. www.akamai.com/products/serverless-computing-edgeworkers
10Akamai’s own llms.txt index for its technical documentation, an example of the static Markdown approach in use by Akamai itself. techdocs.akamai.com/llms.txt
11The per-product llms.txt for EdgeWorkers documentation; append .md to any page URL there for its Markdown version. techdocs.akamai.com/edgeworkers/llms.txt
12Cloudflare’s documentation of its native feature, the reference for the output format and headers this page recommends matching. developers.cloudflare.com/fundamentals/reference/markdown-for-agents/
13The Content Signals framework behind the content-signal header. contentsignals.org/
14MDN’s explanation of HTTP content negotiation and the Accept header. developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Content_negotiation
15The llms.txt proposal, including the recommendation to publish .md versions of pages and the Link header relations used in Route 2. llmstxt.org/
16A runtime-agnostic HTML to Markdown converter with content extraction and token estimation, one option for the conversion step. jsr.io/@markdown-for-agents/core
17Python middleware for Flask, Django and FastAPI that converts HTML to Markdown on Accept: text/markdown, an example of Route 3. pypi.org/project/markdown-agents/
18A developer’s account of publishing static .md versions of pages rather than converting at the edge, with the reasoning. flaviocopes.com/serving-markdown-to-ai-agents/
19BytePlus CDN’s documentation of its own edge HTML to Markdown feature, showing the pattern spreading to other CDNs. docs.byteplus.com/es/docs/byteplus-cdn/html-to-md_en-us

Markdown for Agents on Akamai: serving to AI agents / LLMs.