Build Log: Generating llms-full.txt, Markdown Page Copies and a Link Header From One Docs Folder

307cf805 files changed, +42

Commit 1: the docs folder the build reads

Every page is a Markdown file with a small front matter block. The build needs two fields, title and description, because the llms.txt proposal makes every entry a link with an optional note after a colon, and the description becomes that note. Two more fields place the page: section names the H2 it goes under, order sorts it there.

docs/
  index.md               Quickstart          section: Docs
  guides/retries.md      Retries and backoff section: Docs
  guides/migrating.md    Migrating from v1   section: Optional
  guides/internal.md     Internal runbook    llms: false
  reference/api.md       REST API reference  section: Reference

---
title: Retries and backoff
description: How failed jobs are retried, and how to change the backoff.
section: Docs
order: 2
---
# Retries and backoff

Failed jobs retry 5 times with exponential backoff by default.

The internal runbook carries llms: false. A docs folder usually holds pages you do not want an agent to quote, and the switch belongs in the page, next to the text it protects, not in a list somewhere else that someone forgets to update.

a46e8ca1 file changed, +51

Commit 2: one script, three outputs

build-llms.mjs walks docs/, skips pages marked llms: false, and stops with an error when a page has no title or description, so a missing note fails the build instead of shipping an empty link. For each page it writes three things.

• A Markdown copy at the page URL plus .md. The proposal asks for a clean Markdown version "at the same URL as the original page", either with .md appended (page.html.md) or with the extension replaced (page.md), and index.html.md for a URL that ends in a slash. The script uses the appended form, so /docs/ gets /docs/index.html.md.
• A line in llms.txt under the H2 its section names, linking to that Markdown copy, not to the HTML page. The proposal says the links "should therefore point to LLM-friendly content".
• A block in llms-full.txt with the page title, its source URL and the full text. This file is not in the proposal. It is a convention documentation platforms ship: Mintlify generates one that "combines your entire documentation site into a single file", each page as its title, source URL, description and content.

// build-llms.mjs: docs/**/*.md -> public/llms.txt, public/llms-full.txt, public/**/*.md
import { readdir, readFile, writeFile, mkdir } from 'node:fs/promises'
import { join, relative, dirname } from 'node:path'

const SITE = 'https://example.com'
const DOCS = 'docs'
const OUT = 'public'
const ORDER = ['Docs', 'Reference', 'Optional']

async function walk(dir) {
  const out = []
  for (const e of await readdir(dir, { withFileTypes: true })) {
    const p = join(dir, e.name)
    if (e.isDirectory()) out.push(...(await walk(p)))
    else if (e.name.endsWith('.md')) out.push(p)
  }
  return out
}

function parse(src) {
  const m = src.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/)
  if (!m) throw new Error('missing front matter')
  const meta = Object.fromEntries(
    m[1].split('\n').map((l) => [l.slice(0, l.indexOf(':')).trim(), l.slice(l.indexOf(':') + 1).trim()]),
  )
  return { meta, body: m[2].trim() }
}

const pages = []
for (const file of (await walk(DOCS)).sort()) {
  const { meta, body } = parse(await readFile(file, 'utf8'))
  if (meta.llms === 'false') continue
  if (!meta.title || !meta.description) throw new Error(`${file}: title and description are required`)
  const rel = relative(DOCS, file)
  const html = rel === 'index.md' ? '/docs/' : `/docs/${rel.replace(/\.md$/, '.html')}`
  const md = html.endsWith('/') ? `${html}index.html.md` : `${html}.md`
  pages.push({ ...meta, order: Number(meta.order ?? 99), html, md, body })
  await mkdir(dirname(join(OUT, md)), { recursive: true })
  await writeFile(join(OUT, md), body + '\n')
}

const head = `# Example Queue\n\n> A job queue for Node.js with retries, schedules and a REST API.\n\nUse the Markdown links below; every docs page also exists at its URL plus .md.\n`
const sections = ORDER.map((s) => {
  const list = pages.filter((p) => p.section === s).sort((a, b) => a.order - b.order)
  return list.length ? `\n## ${s}\n\n` + list.map((p) => `- [${p.title}](${SITE}${p.md}): ${p.description}`).join('\n') + '\n' : ''
}).join('')
await writeFile(join(OUT, 'docs/llms.txt'), head + sections)

const rank = (p) => ORDER.indexOf(p.section) * 100 + p.order
const full = [...pages].sort((a, b) => rank(a) - rank(b)).map((p) => `## ${p.title}\nSource: ${SITE}${p.html}\n\n${p.body.replace(/^# .*\n+/, '')}\n`).join('\n---\n\n')
await writeFile(join(OUT, 'docs/llms-full.txt'), `# Example Queue\n\n${full}`)
console.log(`${pages.length} pages, ${sections.split('\n- ').length - 1} links`)

The files go to public/docs/, not the site root. The v2 proposal says a file "covers the URLs under its path" and that "the most specific one" applies, so /docs/llms.txt speaks for the docs only and leaves the marketing pages alone. Running it printed 4 pages, 4 links: five files in, the private one out.

5f682591 file changed, +17

Commit 4: a check that fails the build

The proposal says the format is precise enough to read "using standard programmatic-based tools", so a few lines of regex can enforce it. check-llms.mjs tests what the Format section asks for: the H1 name comes first, there is exactly one H1, no heading below H2, every list item under an H2 is [name](url) with an optional : notes, and every linked Markdown file exists on disk.

// check-llms.mjs: the shape checks from the llms.txt v2 format section
import { readFile } from 'node:fs/promises'
const [file, base = 'public'] = process.argv.slice(2)
const lines = (await readFile(file, 'utf8')).replace(/^/, '').split('\n')
const fail = (m) => { console.error('FAIL', m); process.exitCode = 1 }
if (!/^# \S/.test(lines[0])) fail('line 1 must be the H1 name')
if (lines.filter((l) => /^# /.test(l)).length !== 1) fail('exactly one H1')
let inList = false
for (const [i, l] of lines.entries()) {
  if (/^### /.test(l)) fail(`line ${i + 1}: only H1 and H2 headings`)
  if (/^## /.test(l)) inList = true
  if (inList && l.startsWith('- ') && !/^- \[[^\]]+\]\(https?:\/\/[^)\s]+\)(: .+)?$/.test(l)) fail(`line ${i + 1}: list item is not [name](url): notes`)
}
for (const m of lines.join('\n').matchAll(/\]\(https:\/\/example\.com(\/[^)]+)\)/g)) {
  try { await readFile(base + m[1]) } catch { fail(`missing file for ${m[1]}`) }
}
console.log(process.exitCode ? 'llms.txt has problems' : 'llms.txt OK')

On the real output it printed llms.txt OK and exited 0. To prove it can fail, I broke a copy by hand: one ## Reference turned into ### Reference and one link renamed to a file that does not exist.

$ node check-llms.mjs bad.txt; echo "exit=$?"
FAIL line 12: only H1 and H2 headings
FAIL missing file for /docs/guides/retry.html.md
llms.txt has problems
exit=1

Run it right after the build in CI and a renamed page can no longer leave a dead link in the file agents read first.

3b27b191 file changed, +17

Commit 5: tell agents where the Markdown is

v2 of the proposal, published in August 2026, added discovery. Its changes page says the most common request was "given a page, how does an agent find its markdown version, or the llms.txt file that covers it, without guessing?" The answer is two standard link relations: rel="alternate" type="text/markdown" for the page's Markdown copy and rel="describedby" for the llms.txt file that covers it, sent as HTML <link> elements or as an HTTP Link header. The header also works on the Markdown files themselves, and the proposal notes it can be set in server or CDN configuration without touching a page.

// serve.mjs: static server that adds the llms.txt v2 Link header
import { createServer } from 'node:http'
import { readFile } from 'node:fs/promises'
createServer(async (req, res) => {
  const path = req.url.split('?')[0]
  const file = 'public' + (path.endsWith('/') ? path + 'index.html.md' : path)
  try {
    const body = await readFile(file)
    const isMd = /\.(md|txt)$/.test(file)
    const headers = { 'Content-Type': isMd ? 'text/markdown; charset=utf-8' : 'text/html; charset=utf-8' }
    if (path.startsWith('/docs/') && !path.endsWith('llms.txt')) {
      const md = path.endsWith('.md') ? path : (path.endsWith('/') ? path + 'index.html.md' : path + '.md')
      headers.Link = `<${md}>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"`
    }
    res.writeHead(200, headers).end(body)
  } catch { res.writeHead(404).end('not found') }
}).listen(4791, '127.0.0.1')

This local server stands in for that config: it serves the build output and adds the header to every docs URL except the llms files. The response, captured with curl -sI:

$ curl -sI http://127.0.0.1:4791/docs/guides/retries.html.md
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Link: </docs/guides/retries.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

Hosted platforms do not all use these names. Mintlify's docs show its Link header with rel="llms-txt" and rel="llms-full-txt", plus an X-Llms-Txt header, and it serves both files at /.well-known/ too. If you build your own, the relations the proposal names are the ones to send.

public/output, not committed

The file agents get

683 bytes for llms.txt, 513 for llms-full.txt, four .md copies. A real docs set is larger, and the index can outgrow a context window: Mintlify caps its generated index at 100,000 characters and moves groups of links into extra index files past that, with no such limit on llms-full.txt.

# Example Queue

> A job queue for Node.js with retries, schedules and a REST API.

Use the Markdown links below; every docs page also exists at its URL plus .md.

## Docs

- [Quickstart](https://example.com/docs/index.html.md): Install the CLI and send your first job in five minutes.
- [Retries and backoff](https://example.com/docs/guides/retries.html.md): How failed jobs are retried, and how to change the backoff.

## Reference

- [REST API reference](https://example.com/docs/reference/api.html.md): Every endpoint, its parameters and its error codes.

## Optional

- [Migrating from v1](https://example.com/docs/guides/migrating.html.md): Breaking changes between v1 and v2.