The /llms.txt file is a proposal to standardize a method for providing LLMs with concise, expert-level information about a website. It addresses the limitations of LLM context windows by offering a dedicated markdown file containing background information, guidance, and links to detailed documentation. The format is designed to be both human and machine readable, enabling fixed processing methods. The proposal includes generating markdown versions of existing HTML pages (appending .md to the URL). This initiative aims to improve LLM performance in various applications, from software documentation to complex legal analysis, and is already being implemented in projects like FastHTML and nbdev.
