The /llms.txt file is a proposal to standardize a method for providing LLMs with concise, expert-level information about a website. It addresses the limitations of LLM context windows by offering a dedicated markdown file containing background information, guidance, and links to detailed documentation. The format is designed to be both human and machine readable, enabling fixed processing methods. The proposal includes generating markdown versions of existing HTML pages (appending .md to the URL). This initiative aims to improve LLM performance in various applications, from software documentation to complex legal analysis, and is already being implemented in projects like FastHTML and nbdev.
Vercel proposes using
<script type="text/llms.txt"> to include inline instructions for LLMs directly in HTML responses, particularly for access control and agent navigation.
<pre>
<script type="text/llms.txt">
## Note to agents accessing this page:
This page requires authentication to access. Automated agents should use a
Vercel authentication bypass token to access this page.
The easiest way to get a token is using the get_access_to_vercel_url or ...
</script>
</pre>
<script type="text/llms.txt"> to include inline instructions for LLMs directly in HTML responses, particularly for access control and agent navigation.
<pre>
<script type="text/llms.txt">
## Note to agents accessing this page:
This page requires authentication to access. Automated agents should use a
Vercel authentication bypass token to access this page.
The easiest way to get a token is using the get_access_to_vercel_url or ...
</script>
</pre>
Google’s John Mueller downplayed the usefulness of LLMs.txt, comparing it to the keywords meta tag, as AI bots aren’t currently checking for the file and it opens potential for cloaking.
