Agent-Ready Web doesn't replace existing web standards—it extends and enhances them. ARW builds on top of robots.txt, sitemap.xml, llms.txt, and RFC 8615 to create a complete infrastructure layer for AI agents.
ARW sits at the top of the web infrastructure stack, building on established standards.
What humans see and interact with
How search engines discover content
Where site-wide metadata lives (RFC 8615)
How agents understand, attribute, and act
Key Insight: ARW is the top layer that defines what's inside and how agents act. The layers below define where content lives and who can access it.
ARW complements existing standards by adding semantic understanding and operational capabilities
| Feature | robots.txt | sitemap.xml | llms.txt | ARW |
|---|---|---|---|---|
| Primary Purpose | Access control for crawlers | URL discovery & indexing | AI-readable content manifest | End-to-end agent interoperability |
| Scope | Crawler permissions | Site structure | Basic content discovery | Discovery → Semantics → Actions → Protocols |
| Format | Plain text | XML | Markdown or YAML | YAML + JSON schemas |
| Content Awareness | None | URLs only | Human-readable summaries | Rich content graph with chunks |
| Actionability | None | None | None | ✅ OAuth actions (add_to_cart, etc.) |
| Policy Control | Disallow/Allow crawlers | None | Implicit | ✅ Explicit training/inference policies |
| Observability | None | None | None | ✅ AI-* header namespace |
| Interoperability | None | None | None | ✅ Bridges MCP / ACP / A2A |
| Conformance Levels | None | None | None | ✅ ARW-1 → ARW-4 progressive |
Start with llms.txt and progressively enhance to full ARW conformance. No need to rewrite everything.
Basic content manifest
Add YAML structure + .well-known files
Add .llm.md views + chunks + AI-* headers
Add OAuth-enforced actions
Add MCP / ACP / A2A support
ARW adds .llm.md views alongside HTML, doesn't replace it
OpenGraph, Schema.org, meta tags all remain
ARW reads these for freshness data
OAuth actions use your current auth system
ARW adds OAuth wrapper, keeps your logic
ARW is designed for progressive enhancement. You don't replace llms.txt—you enhance it with YAML structure. You don't replace HTML—you add .llm.md views. You don't replace your API—you add OAuth enforcement. Everything is additive.
Six novel capabilities that transform basic discovery into full agent interoperability
First-class content type (text/x-llm+markdown) with chunk addressability and 85% token reduction
Track which agents access what content and how they use it
Let agents complete transactions with user consent
Reference specific sections of content with stable IDs
Explicit training/inference rules with attribution requirements
Bridge to MCP, ACP, and A2A for agent-to-agent communication
robots.txt and sitemap.xml are like the yellow pages of the web—they tell you where things are and whether you can access them.
llms.txt adds a table of contents—a human-readable guide to what's important.
ARW is the API contract for the agentic web—describing not just where to look, but what you can do, how to do it safely, and under what terms.
Both are valid. Both are useful. ARW just takes you further when you need it.