AI crawlers, robots.txt and llms.txt: a startup guide

The mistake that costs founders the most is a single rule for "AI bots". A blanket block written to keep your content out of model training also removes you from the search index that ChatGPT or Claude consult when a buyer asks for a tool like yours.

The vendors split these jobs on purpose and give each one its own token. OpenAI states that "each setting is independent of the others".^[1] Perplexity says the same about its two agents.^[3] The useful question is which of the three jobs you want done on your pages.

RFC 9309, the 2022 standard for the Robots Exclusion Protocol, is blunt: "These rules are not a form of access authorization."^[13] A robots.txt file is a published preference that well-behaved crawlers choose to follow. Anthropic says its bots honour it.^[2] Amazon says automated crawling from its listed agents respects it.^[11]

The user-triggered fetchers are the documented exception. OpenAI, Perplexity, Meta and Amazon each say their user-initiated agent may not follow robots.txt, on the reasoning that a person asked for that page.^{[1][3][10][11]} Anthropic is the outlier: it documents Claude-User as blockable through robots.txt.^[2] To keep a page from every fetcher, use a login. A firewall rule is real enforcement too, but Anthropic warns that IP blocking can stop its bot reading your robots.txt at all.^[2]

One honest caveat on "refuse training": no vendor document we opened says that allowing a training crawler makes a model more likely to mention you, and none says that blocking one makes it less likely. Vendors only document what the token controls. Choose on what you are comfortable with.

Three things in Policy 2 are judgement calls, so adjust them. Google-Extended also governs grounding in Gemini, not only training, so refusing it has a search-like cost inside Gemini products.^[4] CCBot feeds an open dataset used for research as well as by AI developers.^[9] And Amazonbot is described as improving Amazon products, with training as a possible use, so blocking it is broader than a pure training opt-out.^[11]

Serve the file at the root of each host. Anthropic notes that an opt-out has to be repeated on every subdomain you want covered.^[2]

Jeremy Howard published the proposal on September 3, 2024 at llmstxt.org. The page was revised on August 10, 2026 and now describes itself as "v2 of the proposal".^[12] It asks sites to publish a markdown file that offers "brief background information, guidance, and links to detailed markdown files". The only required element is an H1 with the site or project name; a blockquote summary and H2 sections of links are optional.

It does a different job from robots.txt. The proposal says robots.txt tells automated tools what access is acceptable, while llms.txt is "used on demand, when an agent needs information about a topic while assisting a user".^[12] It is a map, written for inference time. It cannot block or allow a crawler.

Our view: publish one if you have developer docs. It is a small job, the labs publish them for their own docs, and it gives a coding agent, or anyone pasting context into an assistant, a clean starting point. On a five-page marketing site it is harmless and low priority. Either way it is not a ranking lever. CodeTrendy publishes an llms.txt of its own as a markdown index of the site.

When is none of this worth your time? If your site is a week old with a handful of pages, a default robots.txt with a sitemap line is enough: with no AI rules, every documented bot is already allowed. Come back to this when the logs show real volume, or when you publish something you would mind seeing reused.

Being crawlable is only the first condition for being mentioned; what an assistant says depends on what the open web says about you. For places where a product can be described publicly, see where to promote a startup; a profile on CodeTrendy is one such page, and it promises no traffic, ranking or AI citation (plans are on pricing).