# llms.txt, robots.txt, the sitemap and the agent card

> llms.txt, robots.txt, a sitemap and an agent card tell an agent what a site is, which paths it may fetch, and which URL is the page.

Wake publishes all four on its own origin. The scan checks them on yours.

## llms.txt

`/llms.txt` is a Markdown index. One heading, one blockquote that says what the site is in a sentence, then a list of links. Each link gets a short reason.

`/llms-full.txt` is that index with the page text inlined, for an answer engine that will quote one file instead of crawling ten.

Serve both as `text/markdown`. Link them from the HTML with `rel="alternate"`.

## robots.txt

Allow the public pages. Disallow desks, sign-in and the API.

A named `User-agent` group does not inherit the `User-agent: *` group. The most specific group is the whole policy for that bot. If GPTBot or OAI-SearchBot has its own group that only says `Allow: /`, that bot is not bound by the Disallow lines written under `*`. Put every bot in one group that shares one set of rules, or repeat the allows and the disallows in each group.

Allowing a bot to read the docs is a separate choice from allowing it to train. OAI-SearchBot and ChatGPT-User fetch a page because a person asked a question. GPTBot is the training crawler. Name both if you want to be cited and you accept training. Name only the search bots if you do not.

## Sitemap

`/sitemap.xml` lists canonical URLs that return 200 and are meant to be indexed. One URL per page. Leave out the duplicate that redirects, or that serves the same document under another path. `lastmod` is the day the page text changed, not the day the sitemap was generated.

Do not list hop redirects. A pinged redirect is how a search crawler arrives. It is not how an agent that can act arrives.

## Agent card

`/.well-known/agent-card.json` names the service, what it does, and the URL of the docs. A client reads it before it calls a tool.

## The page itself

The first HTML response has to contain the words. A document that is an empty `<div id="root">` is invisible to a client that does not run JavaScript. Put the answer in the HTML. Hide that block with CSS only after the app has rendered the same answer.

A doc that agents will quote should answer `Accept: text/markdown` with the source, and `Accept: text/html` with one `h1`, a title, a description and a canonical link.

## Check

[/wake/start](/wake/start) fetches a public hostname the way GPTBot, ClaudeBot, PerplexityBot, a script and a browser would, and reports which of these are missing or blocking. No account. Why those fetches never show up in Google Analytics is [Why analytics cannot see AI agents](/wake/docs/analytics).
