llms.txt is a proposed, plain-Markdown file placed at a website’s root (yoursite.com/llms.txt) that curates and links to the pages most useful for large language models. It was proposed by Jeremy Howard of Answer.AI in September 2024, and it is a community proposal, not an official standard that AI providers are confirmed to read.
If your team runs a career site and cares about being cited by AI assistants like ChatGPT, Gemini, Perplexity, or Claude, llms.txt is worth understanding. But it is easy to overstate what it does. This guide explains the proposal accurately, contrasts it with files you already know, and gives practical, honest advice.
What Problem Does llms.txt Try To Solve?
The premise is simple. Modern web pages are cluttered with navigation, scripts, and markup that make it harder for a model to find the substance quickly, and context windows are limited. The proposal suggests a curated Markdown file that points models straight to your most important content in a clean, structured form.
Per the specification at llmstxt.org, the file follows a fixed shape: a single H1 with the project or site name, a blockquote summary, optional descriptive prose, and then H2 sections containing bulleted lists of links with short descriptions. An optional companion file, llms-full.txt, expands those linked pages into one long Markdown document a model could ingest in a single fetch.
How is it different from robots.txt and sitemap.xml?
These files are often confused. They solve different problems and are not interchangeable.
| File | Purpose | Format | Status | Officially used by major crawlers/AI? |
|---|---|---|---|---|
| robots.txt | Tells crawlers which paths they may or may not access | Plain text directives | Long-established web convention | Yes, widely honored by search and AI crawlers |
| sitemap.xml | Lists URLs to help crawlers discover and prioritize pages | XML | Supported standard | Yes, used by Google and other search engines |
| llms.txt | Curates and summarizes key content specifically for LLMs | Markdown | Community proposal (2024) | Not confirmed by any major AI provider |
The key distinction: robots.txt controls access and sitemap.xml aids discovery, while llms.txt aims to aid comprehension. But unlike the first two, llms.txt has no confirmed consumer among the major AI platforms.
Do AI Providers Actually Read llms.txt?
This is where honesty matters. As of this writing, no major AI provider has publicly confirmed that it consumes llms.txt as part of crawling, indexing, or answer generation.
Google has been the most explicit. Its Search Central documentation states that you do not need to create special machine-readable files or Markdown to appear in Google Search, including its generative AI features, because Google Search does not use them. Google Search Advocate John Mueller has separately noted that, as far as he is aware, none of the AI services have said they use llms.txt, and that server logs show they do not even request the file.
Independent analyses have reached the same conclusion. In a widely cited July 2025 review, Ahrefs concluded that no major LLM provider currently supports llms.txt, naming OpenAI, Anthropic, and Google specifically.
What is true is that some developer-focused companies publish llms.txt files for their own documentation, and there is genuine interest in the idea for agent and tool-use scenarios where a model is explicitly pointed at a file. Overall adoption, however, remains a very small fraction of the web. Treat any claim that a specific AI product “reads your llms.txt” as unverified unless that provider says so directly.
What Should Recruiting and Career-Site Teams Do?
The upside of publishing llms.txt is low cost and low risk. The realistic downside is spending time you could invest in things that are proven to help AI visibility. Here is a measured approach.
Prioritize the fundamentals first
What actually helps AI assistants understand and cite your roles and employer content is the same work that helps traditional and AI search: clean, crawlable HTML, accurate job data, clear structure, and trustworthy, well-organized employer content. Structured data through valid JobPosting schema remains the documented, provider-supported way to describe jobs to machines. Start there. Our GEO and AEO for recruitment guide and the ultimate guide to career sites cover these foundations in depth.
Treat llms.txt as an experiment, not a strategy
If you have the resources, publishing a clean llms.txt at your root is a reasonable, low-effort experiment. Keep it accurate: an H1 with your employer brand, a one-line summary of who you hire and where, and curated links to your careers homepage, key job categories, benefits, and culture pages. Do not expect measurable ranking or citation gains today, and do not remove or deprioritize proven work to build it.
Do not use it to hide or duplicate content
llms.txt does not replace robots.txt for access control, and it should not contain content that contradicts your live pages. A model or reviewer that fetches both should find them consistent.
For teams thinking about how AI assistants surface employer brands more broadly, our ultimate guide to AI in recruiting provides useful context on the shifting landscape.
Frequently Asked Questions
What is llms.txt in simple terms?
It is a proposed Markdown file at a site’s root that lists and summarizes the pages most useful for large language models, intended to give them clean, structured access to key content.
Who created llms.txt?
It was proposed by Jeremy Howard of Answer.AI in September 2024. The specification is published at llmstxt.org.
Is llms.txt an official standard?
No. It is a community proposal, not a ratified standard, and it has no W3C or IETF working group behind it.
Do ChatGPT, Gemini, or Claude read llms.txt?
No major AI provider has publicly confirmed that it consumes llms.txt for answers or crawling. Google has stated its Search systems, including AI features, do not require or use such files.
Will adding llms.txt improve my AI search rankings?
There is no evidence today that it does. Focus on proven fundamentals like crawlable content and valid JobPosting structured data first.
Should my career site have an llms.txt file?
Optionally, as a low-cost experiment, provided it is accurate and does not divert effort from proven optimizations. It is not a substitute for robots.txt, sitemaps, or structured data.
