llms.txt for agents

docs.page generates llms.txt from public Git docs so agents can read without scraping HTML.

docs.page auto-generates llms.txt (and llms-full.txt) from the same public Git Markdown that powers your site, so coding agents can discover and read your docs without scraping HTML or maintaining a separate export. Sidebars and HTML search still work for humans; agents get a compact, machine-readable index in one HTTP request.

This is the canonical page for using llms.txt with docs.page. For generation rules, caching, and response headers, see How llms.txt is generated.

Before you begin

  • A published site on a public GitHub repository (see Agent-ready docs)
  • Your {owner} and {repo} slugs from the production URL

Overview

Every public docs.page site exposes two auto-generated markdown exports that follow the llms.txt convention:

  • llms.txt: a lightweight index with site title, description, and a Docs section listing each page as a markdown link with its title, summary, and canonical URL
  • llms-full.txt: the same header, then full MDX source for every page (title, Source: URL, and raw content in a fenced block)

Neither export requires a docs.json toggle or build step. docs.page generates both from the same GitHub source as your live site. Push an update and the exports reflect it on the next request.

Production URLs:

text
https://docs.page/{owner}/{repo}/llms.txt
https://docs.page/{owner}/{repo}/llms-full.txt

Branch and pull request previews use the same ~ref segment as page URLs. Custom domains and vanity subdomains serve the same paths on your public origin. See How llms.txt is generated for generation rules, caching, and URL variants.

How it works

What agents get from llms.txt

llms.txt is a map of your docs (titles, summaries, and canonical links) without downloading full page source. Example shape:

markdown
# Acme Handbook

Product documentation for the Acme platform.

## Docs

- [Getting started](https://docs.page/acme/handbook/getting-started): Install the SDK and make your first API call.
- [Authentication](https://docs.page/acme/handbook/authentication): API keys, OAuth, and token refresh.

Agents and integrators often add this URL to a system prompt, tool manifest, or scheduled ingestion job. The tool fetches the index first, then follows links to pages relevant to the user's question.

What agents get from llms-full.txt

llms-full.txt bundles full MDX source for every page in one response. Use it when an agent cannot follow links or prefers a single bulk download over many HTTP requests.

The tradeoff is size: large sites produce large responses. Prefer llms.txt when discovery alone is enough.

Agent needUse
Map of pages: titles, summaries, linksllms.txt
Full source in one downloadllms-full.txt
Per-page reads after discoveryllms.txt, then follow links, or use MCP server
One page's raw source in the browserAppend .md to the page URL, or choose View markdown in the page action menu

How agents use exports in practice

Typical workflows:

  1. Prompt context: include the llms.txt URL so the model knows where to find your catalog
  2. Scheduled ingestion: a job fetches llms.txt or llms-full.txt and loads content into a vector store
  3. On-demand fetch: an agent curls the index at the start of a session, then retrieves individual pages by URL
  4. From any page: a request for a documentation URL with Accept: text/markdown returns that page as markdown. Frontmatter stays first. The page body starts with a documentation index that links to llms.txt. The x-llms-txt header is the same absolute URL. See HTTP endpoints: Agent markdown

RAG pipelines parse the Docs links from llms.txt and fetch pages selectively. One-shot pipelines ingest llms-full.txt when a single markdown blob is simpler than many requests.

Check yours

bash
curl -sS "https://docs.page/{owner}/{repo}/llms.txt" | head

Expect markdown with a Docs section listing your pages.

For branch previews, add ~{ref} after the repository name (the same rules as branch preview):

bash
curl -sS "https://docs.page/{owner}/{repo}~feature-branch/llms.txt" | head

For very large repositories, the response may include an x-docs-page-tree-truncated: 1 header, meaning the export can be incomplete. Use MCP server to read individual pages instead.

Check one page

To verify raw source for a single page without fetching the full site index, open View markdown from the page header menu, or append .md to the page URL:

text
https://docs.page/{owner}/{repo}/getting-started.md

See HTTP endpoints: Raw markdown for path rules.

Related