Discovery Infrastructure for AI Agents: A Comprehensive Guide to llms.txt, agents.json, OpenAPI, and Semantic HTML Patterns

Course: AI Agent Systems Engineering Level: Intermediate–Advanced Estimated Study Time: 45–60 minutes Prerequisites: Basic understanding of HTTP, REST APIs, and HTML; familiarity with JSON and Markdown

Introduction & Learning Objectives

AI agents operating autonomously across the web face a fundamental problem: they cannot reliably discover what a site offers, what actions are permitted, what data is available, or how to interact with services — unless that information is explicitly structured for machine consumption.

Discovery infrastructure is the set of conventions, file formats, and markup patterns that solve this problem. It is the difference between an agent that must scrape, guess, and hallucinate capabilities versus one that reads a machine-authoritative declaration of what exists and what is allowed.

By the end of this lesson, you will be able to:

Define discovery infrastructure and explain why it is architecturally necessary for autonomous AI agents
Implement llms.txt files that give language models structured, prioritized access to site content
Author agents.json manifests that declare agent-accessible capabilities, permissions, and endpoints
Write and expose OpenAPI specifications that enable zero-shot API integration by agents
Apply semantic HTML patterns that make web content simultaneously human-readable and machine-parseable
Identify integration patterns, common failure modes, and emerging standards in this space

Core Concepts: What is Discovery Infrastructure?

Definition

Discovery infrastructure is the collection of machine-readable artifacts, conventions, and protocols that allow AI agents to:

Find what resources, APIs, and capabilities exist at a given domain
Understand the semantics of those resources without prior training on them
Determine what actions are permitted, rate-limited, or prohibited
Execute interactions correctly on the first attempt, without trial-and-error

Why Existing Infrastructure Is Insufficient

Existing Mechanism	Limitation for AI Agents
`robots.txt`	Specifies crawl permissions only; no capability semantics
`sitemap.xml`	Lists URLs; no content type, action, or permission metadata
HTML `<meta>` tags	Inconsistently implemented; no standardized agent vocabulary
OpenGraph / Schema.org	Designed for social sharing and search; not agent action semantics
API documentation (prose)	Requires NLP interpretation; ambiguous; not machine-executable

The Discovery Stack

A complete discovery infrastructure operates at four layers:

Layer 4: Semantic Content     → Semantic HTML, microdata, JSON-LD
Layer 3: API Contracts        → OpenAPI, AsyncAPI, GraphQL SDL
Layer 2: Capability Manifests → agents.json, llms.txt
Layer 1: Protocol Conventions → HTTP headers, well-known URIs, CORS

Each layer serves a distinct purpose. Agents that can read all four layers can operate with near-zero ambiguity about what a service offers and how to use it.

The Agent Discovery Lifecycle

1. Probe well-known URIs (/.well-known/agents.json, /llms.txt)
2. Parse capability manifest → identify available endpoints and permissions
3. Fetch OpenAPI spec → understand request/response schemas
4. Read semantic HTML → extract structured data from content pages
5. Execute actions → with full schema and permission context

llms.txt: Making AI-Readable Content

What llms.txt Is

llms.txt is a Markdown-formatted file placed at the root of a domain (/llms.txt) that provides language models with a curated, prioritized summary of a site's content and structure. It was proposed by Jeremy Howard (fast.ai) in 2024 as a complement to robots.txt — where robots.txt controls crawl access, llms.txt controls comprehension quality.

The core insight: LLMs have finite context windows. When an agent retrieves a site's content, it may encounter thousands of pages of HTML, JavaScript-rendered content, navigation chrome, and boilerplate. llms.txt solves the signal-to-noise problem by giving the model a curated entry point.

File Structure

# [Site Name]

> [One-sentence description of the site's purpose]

[Optional: 1–3 sentence elaboration on scope, audience, or key capabilities]

## [Section Name]

- [Page Title](URL): [Brief description of what this page contains]
- [Page Title](URL): [Brief description]

## [Another Section]

- [Resource Title](URL): [Description]

## Optional

- [Lower-priority resource](URL): [Description]

Annotated Example

# Acme Payments API

> REST API for payment processing, refunds, and subscription management.

Acme supports one-time charges, recurring billing, and multi-currency 
transactions. All endpoints require OAuth 2.0 bearer tokens.

## Core Documentation

- [Authentication Guide](/docs/auth): OAuth 2.0 flows, token scopes, 
  and refresh procedures
- [Payments API Reference](/docs/api/payments): Create, capture, 
  and void payment intents
- [Webhooks](/docs/webhooks): Event types, payload schemas, 
  and signature verification

## Integration Guides

- [Quickstart](/docs/quickstart): End-to-end integration in 15 minutes
- [Error Handling](/docs/errors): Error codes, retry logic, idempotency keys

## Optional

- [Changelog](/docs/changelog): Version history and breaking changes
- [SDKs](/docs/sdks): Client libraries for Python, Node.js, Ruby, Go

llms-full.txt

Sites may also provide /llms-full.txt — a concatenated, full-text version of all documentation pages in a single file. This allows agents with large context windows to load the entire knowledge base in one request rather than following links.

When to use each:

File	Use Case
`/llms.txt`	Agent needs to navigate to specific information; context is limited
`/llms-full.txt`	Agent needs comprehensive knowledge; context window is large

Implementation Rules

Place at domain root: /llms.txt, not /docs/llms.txt
Use canonical URLs: Absolute URLs preferred; relative URLs acceptable if base is clear
Prioritize by agent utility: Put the most action-enabling content first
Keep descriptions functional: Describe what the page enables, not what it is
❌ "About page for Acme Corp"
✅ "Company background, SLA commitments, and support contact information"
Update on content changes: Stale llms.txt is worse than none — agents may cache it
Serve as text/plain: Avoid HTML wrappers; agents expect raw Markdown

HTTP Headers for llms.txt

Content-Type: text/plain; charset=utf-8
Cache-Control: max-age=86400
X-Robots-Tag: noindex

The noindex header prevents search engines from indexing the file itself while keeping it accessible to agents.

agents.json: Agent Capability Declaration

What agents.json Is

agents.json is a structured JSON manifest, served at /.well-known/agents.json, that declares the agent-accessible capabilities of a domain. Where llms.txt is optimized for language model comprehension, agents.json is optimized for programmatic agent consumption — it is machine-readable first.

The format is analogous to /.well-known/openid-configuration in OAuth: a well-known URI that returns a structured document an agent can parse to understand what the service offers without any prior knowledge.

Core Schema

{
  "schema_version": "1.0",
  "name": "Acme Payments",
  "description": "Payment processing API for e-commerce and SaaS platforms",
  "url": "https://api.acme.com",
  "contact": {
    "email": "api-support@acme.com",
    "url": "https://acme.com/support"
  },
  "capabilities": [
    {
      "id": "payments",
      "name": "Payment Processing",
      "description": "Create, capture, void, and refund payment transactions",
      "endpoint": "https://api.acme.com/v2/payments",
      "spec": "https://api.acme.com/openapi.json",
      "auth": {
        "type": "oauth2",
        "flows": ["authorization_code", "client_credentials"],
        "scopes": ["payments:read", "payments:write"]
      }
    },
    {
      "id": "subscriptions",
      "name": "Subscription Management",
      "description": "Create and manage recurring billing subscriptions",
      "endpoint": "https://api.acme.com/v2/subscriptions",
      "spec": "https://api.acme.com/openapi.json#/paths/~1subscriptions",
      "auth": {
        "type": "oauth2",
        "flows": ["client_credentials"],
        "scopes": ["subscriptions:read", "subscriptions:write"]
      }
    }
  ],
  "agent_policies": {
    "rate_limits": {
      "requests_per_minute": 60,
      "requests_per_day": 10000
    },
    "allowed_agents": "*",
    "prohibited_actions": ["bulk_data_export", "account_deletion"],
    "requires_human_confirmation": ["refunds_above_1000_usd"]
  },
  "llms_txt": "https://acme.com/llms.txt",
  "openapi": "https://api.acme.com/openapi.json"
}

Key Fields Explained

capabilities array: Each capability object represents a discrete, agent-executable function. Agents use this to determine what they can do before attempting any action.

agent_policies: This section is critical for safe agent operation. It declares: - Rate limits the agent must respect - Which agent identities are permitted (can reference specific agent IDs or "*" for all) - Actions that are categorically prohibited - Actions that require human-in-the-loop confirmation before execution

requires_human_confirmation: This field is architecturally significant. It allows service operators to declare which actions are high-stakes enough to require human approval, enabling agents to implement appropriate confirmation flows without hardcoding this logic.

agents.json vs. robots.txt

Dimension	robots.txt	agents.json
Primary purpose	Crawl control	Capability declaration
Format	Custom text format	JSON
Semantics	Allow/disallow URL patterns	Structured capability + permission model
Agent guidance	What not to do	What to do and how
Auth information	None	Full auth flow specification

Serving agents.json

GET /.well-known/agents.json HTTP/1.1
Host: acme.com

HTTP/1.1 200 OK
Content-Type: application/json
Access-Control-Allow-Origin: *
Cache-Control: max-age=3600

The Access-Control-Allow-Origin: * header is important — agents running in browser contexts or cross-origin environments need CORS access to read this file.

OpenAPI Specifications: Standardized API Discovery

Why OpenAPI Is Central to Agent Discovery

OpenAPI (formerly Swagger) is the most mature and widely-adopted standard for describing REST APIs in a machine-readable format. For AI agents, a well-written OpenAPI specification is the difference between:

Without OpenAPI: Agent must infer request structure from documentation prose, examples, or trial-and-error
With OpenAPI: Agent has a complete, executable contract — every endpoint, parameter, schema, auth requirement, and response type is formally specified

OpenAPI 3.1 (the current standard) aligns with JSON Schema Draft 2020-12, making it composable with the broader JSON Schema ecosystem.

OpenAPI Structure for Agent Consumption

openapi: "3.1.0"
info:
  title: Acme Payments API
  version: "2.0.0"
  description: |
    REST API for payment processing. All monetary values are in the 
    smallest currency unit (cents for USD). All timestamps are ISO 8601 UTC.
  x-agent-instructions: |
    When creating payments, always check idempotency_key usage to prevent 
    duplicate charges. Use capture_method: automatic for immediate capture.

servers:
  - url: https://api.acme.com/v2
    description: Production
  - url: https://sandbox.api.acme.com/v2
    description: Sandbox — use for testing; no real charges

paths:
  /payments:
    post:
      operationId: createPayment
      summary: Create a payment intent
      description: |
        Creates a new payment intent. The payment is not charged until 
        captured. Use idempotency_key to safely retry failed requests.
      x-agent-risk-level: medium
      x-requires-human-confirmation: false
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreatePaymentRequest'
            examples:
              basic_charge:
                summary: Simple one-time charge
                value:
                  amount: 2000
                  currency: "usd"
                  payment_method: "pm_card_visa"
                  idempotency_key: "order_123_attempt_1"
      responses:
        '201':
          description: Payment intent created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PaymentIntent'
        '400':
          $ref: '#/components/responses/ValidationError'
        '402':
          $ref: '#/components/responses/PaymentError'

components:
  schemas:
    CreatePaymentRequest:
      type: object
      required: [amount, currency, payment_method]
      properties:
        amount:
          type: integer
          minimum: 50
          description: Amount in smallest currency unit (e.g., cents)
          example: 2000
        currency:
          type: string
          pattern: '^[a-z]{3}$'
          description: ISO 4217 lowercase currency code
          example: "usd"
        payment_method:
          type: string
          description: Payment method ID from Acme.js tokenization
          example: "pm_card_visa"
        idempotency_key:
          type: string
          maxLength: 255
          description: |
            Unique key for safe request retry. Use the same key to safely 
            retry a request without risk of duplicate charges.

Agent-Specific OpenAPI Extensions

Standard OpenAPI does not include agent-specific semantics. Use x- extension fields to add them:

Extension	Purpose	Example Value
`x-agent-instructions`	Free-text guidance for agents using this operation	`"Always verify idempotency_key before retry"`
`x-agent-risk-level`	Classifies operation risk for agent decision-making	`"low"`, `"medium"`, `"high"`, `"critical"`
`x-requires-human-confirmation`	Flags operations needing human approval	`true` / `false`
`x-rate-limit-group`	Groups operations under a shared rate limit	`"write_operations"`
`x-idempotent`	Declares whether the operation is safe to retry	`true` / `false`

OpenAPI Quality Checklist for Agent Consumption

[ ] Every operation has a unique, descriptive operationId
[ ] All parameters have description fields explaining semantics, not just types
[ ] Request body schemas have example values for every field
[ ] All error responses are documented with schema references
[ ] Authentication schemes are fully specified in securitySchemes
[ ] Enum values have descriptions explaining when to use each
[ ] Monetary amounts specify unit (cents vs. dollars) in description
[ ] Timestamps specify format (ISO 8601, Unix epoch) in description
[ ] Idempotency behavior is documented for all write operations
[ ] Sandbox/test server is listed separately from production

Serving OpenAPI Specifications

GET /openapi.json HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json
Access-Control-Allow-Origin: *
Cache-Control: max-age=300

Also consider: - /openapi.yaml for YAML format (more human-readable, same semantics) - Link from agents.json and llms.txt for discoverability - Version the spec URL (/openapi/v2.json) to allow parallel versions

Semantic HTML Patterns: Human & Machine-Readable Content

Why Semantic HTML Matters for Agents

When agents retrieve web pages — for research, data extraction, or action execution — the quality of the HTML determines how accurately they can parse content. Semantic HTML provides structural signals that allow agents to distinguish:

Navigation from content
Headings from body text
Data tables from layout tables
Form fields and their purposes
Prices, dates, addresses, and other typed data

Core Semantic HTML Elements

Document structure:

<header>   <!-- Site/page header; typically navigation and branding -->
<nav>      <!-- Navigation links; agents skip this for content extraction -->
<main>     <!-- Primary page content; agents focus here -->
<article>  <!-- Self-contained content unit (blog post, product, etc.) -->
<section>  <!-- Thematic grouping within main content -->
<aside>    <!-- Supplementary content; lower priority for agents -->
<footer>   <!-- Page footer; typically skip for content extraction -->

Content semantics:

<h1>–<h6>    <!-- Heading hierarchy; agents use for document outline -->
<time>       <!-- Machine-readable dates and times -->
<address>    <!-- Contact information -->
<figure>     <!-- Images with captions -->
<figcaption> <!-- Caption for figure -->
<data>       <!-- Machine-readable value with human-readable display -->
<meter>      <!-- Scalar measurement -->
<output>     <!-- Result of a calculation or user action -->

JSON-LD for Structured Data

JSON-LD (JavaScript Object Notation for Linked Data) embedded in <script type="application/ld+json"> tags is the highest-fidelity method for communicating structured data to agents. It uses Schema.org vocabulary — a shared ontology maintained by Google, Microsoft, Yahoo, and Yandex.

Product page example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Acme Pro Subscription",
  "description": "Full-featured plan with API access and priority support",
  "sku": "ACME-PRO-MONTHLY",
  "offers": {
    "@type": "Offer",
    "price": "99.00",
    "priceCurrency": "USD",
    "priceValidUntil": "2026-12-31",
    "availability": "https://schema.org/InStock",
    "url": "https://acme.com/pricing/pro"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.8",
    "reviewCount": "1247"
  }
}
</script>

Article/documentation page example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "OAuth 2.0 Authentication Guide",
  "description": "Complete guide to authenticating with the Acme API using OAuth 2.0",
  "datePublished": "2025-03-15",
  "dateModified": "2026-04-02",
  "author": {
    "@type": "Organization",
    "name": "Acme Engineering"
  },
  "proficiencyLevel": "Intermediate",
  "dependencies": "OAuth 2.0, HTTP, JSON"
}
</script>

Microdata Patterns

Microdata embeds Schema.org semantics directly in HTML attributes. Less flexible than JSON-LD but useful when the structured data is directly tied to visible content:

<div itemscope itemtype="https://schema.org/FAQPage">
  <div itemscope itemprop="mainEntity" itemtype="https://schema.org/Question">
    <h3 itemprop="name">What payment methods do you accept?</h3>
    <div itemscope itemprop="acceptedAnswer" itemtype="https://schema.org/Answer">
      <p itemprop="text">
        We accept Visa, Mastercard, American Express, and ACH bank transfers.
      </p>
    </div>
  </div>
</div>

ARIA landmark roles provide semantic structure that agents can use to navigate pages efficiently:

<div role="banner">      <!-- Equivalent to <header> -->
<div role="navigation">  <!-- Equivalent to <nav> -->
<div role="main">        <!-- Equivalent to <main> -->
<div role="search">      <!-- Search form region -->
<div role="form">        <!-- Form region -->
<div role="contentinfo"> <!-- Equivalent to <footer> -->

Use native HTML elements where possible; ARIA roles are for cases where semantic HTML elements cannot be used.

The `<time>` Element Pattern

Dates and times are frequently misinterpreted by agents parsing prose. The <time> element eliminates ambiguity:

<!-- Ambiguous for agents: -->
<p>Last updated March 15, 2025</p>

<!-- Unambiguous: -->
<p>Last updated <time datetime="2025-03-15">March 15, 2025</time></p>

<!-- With time and timezone: -->
<time datetime="2025-03-15T14:30:00Z">March 15, 2025 at 2:30 PM UTC</time>

Integration Patterns & Best Practices

Pattern 1: The Discovery Chain

Design your discovery infrastructure so each artifact points to the others, creating a navigable graph:

robots.txt
  └── Sitemap: /sitemap.xml

/.well-known/agents.json
  ├── llms_txt: /llms.txt
  ├── openapi: /openapi.json
  └── capabilities[].spec: /openapi.json#/paths/...

/llms.txt
  ├── Links to key documentation pages
  └── Links to /openapi.json

/openapi.json
  └── info.x-agents-json: /.well-known/agents.json

Pattern 2: Layered Specificity

Provide discovery artifacts at multiple levels of specificity:

Domain level:    /.well-known/agents.json  (all capabilities)
Section level:   /api/llms.txt             (API-specific guidance)
Resource level:  JSON-LD in each page      (per-resource structured data)

Pattern 3: Environment Segregation

Explicitly separate sandbox and production environments in all discovery artifacts:

In agents.json:

{
  "environments": {
    "production": {
      "url": "https://api.acme.com/v2",
      "agents.json": "https://api.acme.com/.well-known/agents.json"
    },
    "sandbox": {
      "url": "https://sandbox.api.acme.com/v2",
      "agents.json": "https://sandbox.api.acme.com/.well-known/agents.json",
      "note": "No real charges; test card numbers available at /docs/testing"
    }
  }
}

Pattern 4: Progressive Disclosure

Structure llms.txt and agents.json to support progressive disclosure — agents with limited context windows get the most important information first:

First 500 tokens: What the service does, primary capability, auth type
Next 1000 tokens: Core API endpoints and their purposes
Remaining: Edge cases, optional features, changelog

Pattern 5: Idempotency Declaration

Explicitly declare idempotency behavior across all artifacts:

In OpenAPI: x-idempotent: true on GET, HEAD, OPTIONS; document idempotency_key on write operations
In agents.json: "idempotent_methods": ["GET", "HEAD"]
In llms.txt: Include a link to idempotency documentation with description "Safe retry patterns for write operations"

Practical Implementation Examples

Minimal Viable Discovery Stack

For a simple SaaS product, the minimum viable discovery infrastructure:

Step 1: /llms.txt

# MyApp

> Project management tool with time tracking and invoicing.

## Key Pages

- [API Overview](/api): REST API for integrating MyApp with other tools
- [Authentication](/api/auth): API key authentication setup
- [Projects API](/api/projects): Create, read, update, delete projects
- [Time Entries API](/api/time): Log and query time entries
- [Invoices API](/api/invoices): Generate and send invoices

## Optional

- [Webhooks](/api/webhooks): Real-time event notifications
- [Rate Limits](/api/rate-limits): Request limits and backoff strategies

Step 2: /.well-known/agents.json

{
  "schema_version": "1.0",
  "name": "MyApp",
  "description": "Project management with time tracking and invoicing",
  "url": "https://myapp.com",
  "capabilities": [
    {
      "id": "projects",
      "name": "Project Management",
      "description": "CRUD operations for projects and tasks",
      "endpoint": "https://api.myapp.com/v1/projects",
      "spec": "https://api.myapp.com/openapi.json",
      "auth": {"type": "api_key", "header": "X-API-Key"}
    }
  ],
  "agent_policies": {
    "rate_limits": {"requests_per_minute": 30},
    "requires_human_confirmation": ["invoice_send", "project_delete"]
  },
  "llms_txt": "https://myapp.com/llms.txt",
  "openapi": "https://api.myapp.com/openapi.json"
}

Step 3: OpenAPI spec — follow the structure in the OpenAPI section above, ensuring every operation has operationId, description, and examples.

Step 4: JSON-LD on key pages — add TechArticle schema to documentation pages, SoftwareApplication schema to the product homepage.

Testing Your Discovery Infrastructure

# Verify llms.txt is accessible and correct content type
curl -I https://yourdomain.com/llms.txt
# Expected: Content-Type: text/plain

# Verify agents.json is accessible with CORS
curl -H "Origin: https://agent.example.com" \
     -I https://yourdomain.com/.well-known/agents.json
# Expected: Access-Control-Allow-Origin: *

# Validate OpenAPI spec
npx @redocly/cli lint openapi.json

# Check JSON-LD validity
# Use: https://validator.schema.org/

# Test agent discovery chain
curl https://yourdomain.com/.well-known/agents.json | \
  jq '.llms_txt, .openapi'
# Should return URLs to both files

Interoperability & Standards Alignment

Relationship to Existing Standards

robots.txt (RFC 9309): agents.json extends, not replaces, robots.txt. robots.txt remains the authoritative source for crawl permissions. agents.json adds capability semantics on top. Agents should respect both.

OpenAPI Initiative: OpenAPI 3.1 is the current standard. The OpenAPI Initiative is working on extensions for AI agent use cases. The x- extension fields described in this lesson are forward-compatible with any future standardization.

Schema.org: JSON-LD with Schema.org vocabulary is the W3C-endorsed approach for structured data. It is supported natively by Google, Microsoft Bing, and Apple's Siri. Using Schema.org ensures your structured data works across both traditional search and AI agent contexts.

W3C Web of Things (WoT): For IoT and physical-world agents, W3C WoT Thing Descriptions provide a similar capability declaration pattern. agents.json is conceptually aligned with WoT Thing Descriptions but optimized for web services rather than IoT devices.

Model Context Protocol (MCP): Anthropic's Model Context Protocol (2024) defines a standard for how AI models connect to external tools and data sources. agents.json can reference MCP server endpoints, making the two standards complementary:

{
  "capabilities": [
    {
      "id": "mcp_server",
      "protocol": "mcp",
      "endpoint": "https://api.acme.com/mcp",
      "description": "MCP server exposing Acme tools to Claude and compatible agents"
    }
  ]
}

OpenAI Plugin Manifest (deprecated): The ChatGPT plugin manifest (ai-plugin.json) was an early attempt at agent capability declaration. It has been superseded by GPT Actions (which use OpenAPI) and is not recommended for new implementations. agents.json is the community-driven successor.

Standards Landscape Summary

Standard	Maintained By	Status	Agent Relevance
robots.txt	RFC 9309	Active	Crawl permissions
OpenAPI 3.1	OpenAPI Initiative	Active	API contracts
Schema.org / JSON-LD	W3C / Schema.org	Active	Structured content
llms.txt	Community (fast.ai)	Emerging	LLM content guidance
agents.json	Community	Emerging	Capability declaration
MCP	Anthropic	Active	Tool integration protocol
W3C WoT	W3C	Active	IoT agent capabilities

Common Pitfalls & Solutions

Pitfall 1: Stale Discovery Artifacts

Problem: llms.txt or agents.json describes endpoints or content that no longer exists. Agents attempt to use deprecated endpoints, receive errors, and may hallucinate alternatives.

Solution: - Automate generation of llms.txt from your CMS or documentation system - Generate agents.json from your OpenAPI spec programmatically - Add discovery artifact validation to your CI/CD pipeline - Set appropriate Cache-Control headers (short TTL for frequently-changing services)

Pitfall 2: Underspecified OpenAPI Schemas

Problem: OpenAPI spec lists endpoints but provides minimal schema detail — no descriptions, no examples, no error documentation. Agents generate incorrect requests.

Solution: - Treat OpenAPI descriptions as executable documentation, not optional metadata - Every description field should answer: "What does this field mean, what are valid values, what are the units?" - Add at least one example per request body schema - Document all error responses, not just 200

Pitfall 3: Missing CORS Headers on Discovery Files

Problem: agents.json and openapi.json are inaccessible to agents running in browser contexts or cross-origin environments due to missing CORS headers.

Solution:

location ~ ^/(\.well-known/agents\.json|openapi\.json|llms\.txt)$ {
    add_header Access-Control-Allow-Origin *;
    add_header Access-Control-Allow-Methods "GET, OPTIONS";
}

Pitfall 4: No Human Confirmation Declarations

Problem: agents.json and OpenAPI spec do not declare which operations require human confirmation. Agents execute high-stakes operations (deletions, large financial transactions, bulk sends) autonomously.

Solution: - Audit all write operations for risk level - Add requires_human_confirmation to agents.json agent_policies - Add x-requires-human-confirmation: true to high-risk OpenAPI operations - Document the threshold (e.g., "transactions above $500 require confirmation")

Pitfall 5: Conflating llms.txt with robots.txt Semantics

Problem: Teams treat llms.txt as a permission file ("if it's not in llms.txt, agents can't access it"). llms.txt is a guidance file, not an access control mechanism.

Solution: - Use robots.txt for access control - Use llms.txt for content prioritization and comprehension guidance - Document this distinction in your internal standards

Pitfall 6: Inconsistent Capability Declarations

Problem: agents.json lists capabilities that are not in the OpenAPI spec, or the OpenAPI spec has operations not referenced in agents.json. Agents get inconsistent pictures of what is available.

Solution: - Generate agents.json capabilities programmatically from OpenAPI tags - Add a validation step that checks agents.json capability endpoints exist in OpenAPI spec - Use a single source of truth (OpenAPI) and derive all other artifacts from it

Future Directions & Emerging Standards

Agent-to-Agent Discovery

Current discovery infrastructure assumes a human-operated service being discovered by an AI agent. Emerging patterns address agent-to-agent discovery — where one AI agent needs to find and delegate to another:

Agent Cards (Google A2A Protocol, 2025): Structured declarations of agent capabilities, analogous to agents.json but for agent-to-agent communication
DID Documents for agent identity: Decentralized identifiers for agents, enabling cryptographic verification of agent identity during discovery

Capability Versioning

As agents become long-running and persistent, they need to handle capability changes gracefully. Emerging patterns include:

Semantic versioning in agents.json with deprecation timelines
Sunset HTTP headers on deprecated endpoints
Change notification webhooks for registered agents

Semantic Capability Matching

Current discovery requires agents to read and interpret capability descriptions. Future systems may use formal ontologies (OWL, RDF) to enable semantic matching — an agent looking for "payment processing" can find services that declare this capability even if they use different terminology.

LLM-Optimized API Design

Beyond documentation, API design itself is evolving for LLM consumption:

Natural language operation IDs: createPaymentForOrder vs. POST /payments
Semantic error messages: Errors that explain what the agent should do next, not just what went wrong
Capability probing endpoints: GET /capabilities?intent=process_refund returns the specific operations needed for a stated intent

Standardization Trajectory

The llms.txt and agents.json formats are community-driven and not yet formal standards. The likely standardization path:

Community adoption and iteration (current phase)
Industry working group formation (IETF or W3C)
RFC or W3C Note publication
Browser and platform native support

Teams implementing these today should expect schema evolution and design their implementations to be version-tolerant.

Key Takeaways & Next Steps

Core Principles

Discovery infrastructure is not optional for production AI agent integrations. Without it, agents operate on guesswork; with it, they operate on contracts.
Each layer serves a distinct purpose. llms.txt guides comprehension. agents.json declares capabilities and permissions. OpenAPI specifies executable contracts. Semantic HTML structures content. None replaces the others.
Machine-readable does not mean human-unreadable. The best discovery artifacts are useful to both human developers and AI agents — they are documentation that happens to be structured.
Permissions and risk declarations belong in the infrastructure. requires_human_confirmation, prohibited_actions, and rate_limits in agents.json are not optional — they are the mechanism by which service operators maintain control over agent behavior.
Generate, don't maintain manually. Discovery artifacts that are manually maintained will drift from reality. Automate generation from your source of truth (OpenAPI spec, CMS, database schema).

Implementation Priority Order

For teams starting from zero:

Week 1: Add llms.txt to your domain root — low effort, immediate value
Week 2: Audit and improve your OpenAPI spec quality (descriptions, examples, error docs)
Week 3: Create /.well-known/agents.json referencing your OpenAPI spec
Week 4: Add JSON-LD to your highest-traffic content pages
Ongoing: Automate artifact generation and add validation to CI/CD

Self-Assessment Questions

Can an AI agent discover all your service's capabilities from /.well-known/agents.json alone?
Does your OpenAPI spec have description and example on every field?
Have you declared which operations require human confirmation?
Are your discovery artifacts served with correct Content-Type and CORS headers?
Do your discovery artifacts cross-reference each other?

References & Additional Resources

Specifications & Standards

llms.txt specification: llmstxt.org — Jeremy Howard's original proposal and community extensions
OpenAPI Specification 3.1: spec.openapis.org/oas/v3.1.0
Schema.org vocabulary: schema.org — Full vocabulary reference for JSON-LD structured data
robots.txt RFC 9309: datatracker.ietf.org/doc/rfc9309
W3C JSON-LD 1.1: w3.org/TR/json-ld11
Model Context Protocol: modelcontextprotocol.io — Anthropic's tool integration standard
Google Agent-to-Agent Protocol: Google DeepMind A2A specification (2025)

Tooling

Redocly CLI: OpenAPI linting and validation — npm install -g @redocly/cli
Schema.org Validator: validator.schema.org — Validate JSON-LD markup
Swagger Editor: editor.swagger.io — Browser-based OpenAPI editing
OpenAPI Generator: openapi-generator.tech — Generate client SDKs and server stubs from OpenAPI specs

Discovery Infrastructure for AI Agents: A Comprehensive Guide to llms.txt, agents.json, OpenAPI, and Semantic HTML Patterns

Discovery Infrastructure for AI Agents: A Comprehensive Guide to llms.txt, agents.json, OpenAPI, and Semantic HTML Patterns

Introduction & Learning Objectives

Core Concepts: What is Discovery Infrastructure?

Definition

Why Existing Infrastructure Is Insufficient

The Discovery Stack

The Agent Discovery Lifecycle

llms.txt: Making AI-Readable Content

What llms.txt Is

File Structure

Annotated Example

llms-full.txt

Implementation Rules

HTTP Headers for llms.txt

agents.json: Agent Capability Declaration

What agents.json Is

Core Schema

Key Fields Explained

agents.json vs. robots.txt

Serving agents.json

OpenAPI Specifications: Standardized API Discovery

Why OpenAPI Is Central to Agent Discovery

OpenAPI Structure for Agent Consumption

Agent-Specific OpenAPI Extensions

OpenAPI Quality Checklist for Agent Consumption

Serving OpenAPI Specifications

Semantic HTML Patterns: Human & Machine-Readable Content

Why Semantic HTML Matters for Agents

Core Semantic HTML Elements

JSON-LD for Structured Data

Microdata Patterns

ARIA Roles for Agent Navigation

The <time> Element Pattern

Integration Patterns & Best Practices

Pattern 1: The Discovery Chain

Pattern 2: Layered Specificity

Pattern 3: Environment Segregation

Pattern 4: Progressive Disclosure

Pattern 5: Idempotency Declaration

Practical Implementation Examples

Minimal Viable Discovery Stack

Testing Your Discovery Infrastructure

Interoperability & Standards Alignment

Relationship to Existing Standards

Standards Landscape Summary

Common Pitfalls & Solutions

Pitfall 1: Stale Discovery Artifacts

Pitfall 2: Underspecified OpenAPI Schemas

Pitfall 3: Missing CORS Headers on Discovery Files

Pitfall 4: No Human Confirmation Declarations

Pitfall 5: Conflating llms.txt with robots.txt Semantics

Pitfall 6: Inconsistent Capability Declarations

Future Directions & Emerging Standards

Agent-to-Agent Discovery

Capability Versioning

Semantic Capability Matching

LLM-Optimized API Design

Standardization Trajectory

Key Takeaways & Next Steps

Core Principles

Implementation Priority Order

Self-Assessment Questions

References & Additional Resources

Specifications & Standards

Tooling

Further Reading

The `<time>` Element Pattern