# Make this site readable and safe for AI agents

Instructions for a coding agent (Claude Code, Codex, Cursor, Copilot, Gemini CLI or any other). A person copied this file into their repository, or pointed you at `https://webmcp-audit.com/agent-instructions.md`. Follow it top to bottom. It is written so that you can finish without asking anything except the three decisions marked ASK.

Goal: after you are done, `https://webmcp-audit.com/` reports every agent-readiness check as passing for the site, and the WebMCP grade (if the site registers tools) is A or B. The catalogue of every check, with the fix and the source document for each, is at `https://webmcp-audit.com/checks`. Fetch it when a check id is unclear.

## 0. Ground rules

- Never invent facts. Everything you write into `llms.txt`, the meta description or JSON-LD must come from the site's own content or from the person. If a fact is missing, leave a `TODO:` and tell them.
- Never put secrets, tokens, internal hostnames or private data into any of these files. They are public.
- Keep files small and plain. A `robots.txt` is a few lines; an `llms.txt` fits on one screen.
- Do not add tracking, frameworks or dependencies to do this. Static files and a few `<meta>` tags are enough.
- Do not delete existing rules in `robots.txt`; add to them and explain each change in your summary.
- If the site is a single-page app that renders everything client-side, say so: the metadata below must be in the server-rendered HTML, not injected by JavaScript later.

## 1. Discover the project

Work out, and write down for the person:

1. The public origin (scheme and host) the site is served from. Look at the deploy config, the README, `CNAME`, `vercel.json`, `netlify.toml`, nginx configs. ASK if you cannot find it.
2. Which directory becomes the web root (`public/`, `static/`, `dist/`, `_site/`, the Next.js `public/` folder, a Django `STATIC_ROOT` and so on) and how root-level files such as `/robots.txt` are served. Framework specifics: Next.js serves `public/robots.txt` as-is or you can add `app/robots.ts`; Astro and Hugo use `public/` and `static/`; a plain nginx vhost serves from `root`.
3. Which page is the entry point a recruiter, customer or agent would open first, and the five to ten pages that matter most (CV, about, projects, pricing, docs, contact).
4. Whether the site already has any of: `robots.txt`, `sitemap.xml`, `llms.txt`, `.well-known/security.txt`, `<meta name="description">`, `<link rel="canonical">`, Open Graph tags, JSON-LD.

## 2. Run the audit first

Get the baseline before changing anything, so the person can see the difference.

Option A, the public service (no install):

```
curl -s -X POST https://webmcp-audit.com/api/audit -H 'Content-Type: application/json' -d '{"url":"https://THE-SITE"}'
# -> {"id":"...","status":"queued","status_url":"/api/audit/<id>"}
curl -s https://webmcp-audit.com/api/audit/<id>            # poll until "status":"done"
curl -s https://webmcp-audit.com/api/audit/<id>/result     # full JSON: result.site.checks, result.site.findings, result.findings, result.grade
```

Limits: five audits per address per hour, three per target site per hour, the same page is served from its last report for an hour. The site must be public; the scanner refuses private and internal addresses.

Option B, locally, if the person has a checkout of the scanner (Node 20+, `cd scanner && npm install` downloads Chrome):

```
node scanner/bin/webmcp-scan.js https://THE-SITE            # human-readable
node scanner/bin/webmcp-scan.js https://THE-SITE --json      # machine-readable
node scanner/bin/webmcp-scan.js https://THE-SITE --check site.llms-txt-missing   # one check
```

Read `result.site.checks` (status per check) and `result.site.findings` (what to fix, by `check_id`). Each `check_id` has an entry at `https://webmcp-audit.com/checks#<check_id>` with the fix and its source.

## 3. Generate the files

Do each of these unless the audit already passes it.

### 3.1 robots.txt (check ids: site.robots-*, site.ai-policy-unstated, site.sitemap-*)

ASK the person one question, with a default: "Should AI companies be allowed to use the site for model training? Default: no preference stated." Then write:

```
User-agent: *
Allow: /

# Search engines and AI search (OpenAI, Anthropic, Perplexity, Google, Bing, Apple)
# and on-demand assistant fetchers (ChatGPT-User, Claude-User, Perplexity-User) are allowed above.
# Training crawlers: uncomment to opt out. Vendors document that this does not affect search or user requests.
# User-agent: GPTBot
# User-agent: ClaudeBot
# User-agent: Google-Extended
# User-agent: Applebot-Extended
# User-agent: CCBot
# Disallow: /

Content-Signal: search=yes, ai-input=yes

Sitemap: https://THE-SITE/sitemap.xml
```

If the answer to the question is "no training", uncomment the block and add `ai-train=no` to the Content-Signal line. Keep every existing `Disallow` the project already had. Never block `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` or `Googlebot` unless the person explicitly wants the site hidden from assistants. Sources: OpenAI `developers.openai.com/docs/bots`, Anthropic `support.claude.com/en/articles/8896518`, Perplexity `docs.perplexity.ai/guides/bots`, Google `developers.google.com/search/docs/crawling-indexing/google-common-crawlers`, Cloudflare Content Signals `blog.cloudflare.com/content-signals-policy/`.

### 3.2 sitemap.xml (check id: site.sitemap-missing)

If the framework can generate one (Next.js `app/sitemap.ts`, Astro `@astrojs/sitemap`, Hugo built-in, Django `sitemaps`), enable that. Otherwise write a static file listing every public page with an absolute URL:

```
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://THE-SITE/</loc></url>
  <url><loc>https://THE-SITE/cv</loc></url>
</urlset>
```

### 3.3 llms.txt (check ids: site.llms-txt-missing, site.llms-txt-malformed)

Format from `llmstxt.org` (Jeremy Howard, Answer.AI). The H1 is the only required line. Write it from the site's real content:

```
# <Site or person name>

> <One or two sentences: who or what this is, where, what someone would come here for.>

## <Section, e.g. CV>
- [<Page title>](https://THE-SITE/<path>): <one line on what is there>

## <Section, e.g. Projects>
- [<Page title>](https://THE-SITE/<path>): <one line>

## Optional
- [<Less important page>](https://THE-SITE/<path>): <one line>
```

Rules: absolute URLs; five to twenty links; the blockquote is the summary an assistant will quote, so write it as you would want to be described; `## Optional` is for pages an agent may skip. If the site has plain-text or markdown versions of pages, link those. Google Search ignores this file (it neither helps nor harms ranking, per Google's June 2026 guidance); Chrome Lighthouse's agentic-browsing audit checks for it; coding agents and assistants read it.

### 3.4 security.txt (check ids: site.security-txt-*)

ASK the person for a security contact (an email or a URL). Write `/.well-known/security.txt` with the two required fields from RFC 9116, Expires under a year ahead:

```
Contact: mailto:security@THE-SITE
Expires: <ISO 8601 date, e.g. 2027-10-01T00:00:00.000Z>
Preferred-Languages: en
```

### 3.5 Page metadata (check ids: site.title-missing, site.description-missing, site.canonical-missing, site.open-graph-missing, site.structured-data-missing, site.lang-missing, site.meta-noindex)

On the entry page and the pages that matter, in the server-rendered `<head>`:

```
<html lang="en">
<title>Jane Doe: backend engineer, Berlin</title>
<meta name="description" content="CV and projects of Jane Doe, senior backend engineer (Go, Postgres).">
<link rel="canonical" href="https://THE-SITE/">
<meta property="og:title" content="Jane Doe">
<meta property="og:description" content="Backend engineer, Berlin">
<meta property="og:image" content="https://THE-SITE/card.png">
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Person","name":"Jane Doe","jobTitle":"Backend engineer","url":"https://THE-SITE/","sameAs":["https://www.linkedin.com/in/..."]}
</script>
```

Use `Organization`, `Product` or `Article` instead of `Person` where that is what the page is. Only state what is visible on the page. Remove any `noindex` from pages that should be found; keep it on status and admin pages. If a CDN or WAF shows a JavaScript challenge to bots (check id: site.page-unreadable), tell the person which setting to change (for Cloudflare: allow verified bots, or an allow rule for the published IP ranges of the fetchers above).

## 4. WebMCP (only if the site has actions an agent should perform)

A CV or a brochure site does not need this. A site with search, booking, ordering or account actions does. Follow `https://webmcp-audit.com/webmcp` (the implementation checklist) and generate the registration with `https://webmcp-audit.com/builder` (its MCP server tab also exports the same tools as a stdio MCP server for agents that do not read a browser tab), or write it by hand:

```js
const mc = document.modelContext;
if (mc && typeof mc.registerTool === 'function') {
  const lifecycle = new AbortController();
  await mc.registerTool({
    name: 'search_products',
    title: 'Search products',
    description: 'Search the product catalogue by keyword and return up to 20 matches.',
    inputSchema: { type: 'object', properties: { query: { type: 'string', maxLength: 120 } }, required: ['query'] },
    annotations: { readOnlyHint: true, untrustedContentHint: true, consequentialHint: false, debugging: false },
    execute: async (input, { signal }) => search(validate(input), signal),
  }, { signal: lifecycle.signal });
}
```

Rules the audit enforces: ASCII names `[A-Za-z0-9_.-]`, 1 to 128 characters; a description that says what the tool does and nothing addressed to the model; a JSON Schema object with `type`, `properties` and `required`; `readOnlyHint` on every tool and `consequentialHint: true` on anything that buys, books, pays, deletes or submits; no password, token or card number as an argument; `exposedTo` with exact origins only, never `*`; HTTPS; no `Permissions-Policy: tools=()`. To make the integration visible to real visitors, register the origin for the WebMCP origin trial and serve the token in an `Origin-Trial` header or `<meta http-equiv="origin-trial">` on every page that registers tools, otherwise the audit reports the deployment as "dark".

## 5. Re-run the audit and report back

Run the audit again (section 2). Then write a short summary for the person:

- Files created or changed, with paths.
- The three decisions you took or still need: training opt-out, security contact, anything marked `TODO:`.
- Before and after: readiness checks passing (x of y), WebMCP grade if any, and the report URL.
- Anything you could not fix from the repository (CDN challenge pages, DNS, a host that strips headers).

Do not stop at "it should work": re-run and paste the numbers.