What is llms.txt?

llms.txt is a proposed standard format for providing website content optimized for LLM consumption. Created to help AI assistants understand documentation, APIs, and open source projects, the file contains structured links and descriptions organized by topic — similar to robots.txt but for language models.

The format was proposed to give AI tools a curated, high-signal entry point into a site rather than forcing them to crawl and interpret arbitrary HTML.

How it works

This tool crawls the provided URL using Cloudflare Browser Rendering, discovers linked pages via sitemaps and internal links, then uses Claude AI to analyze the content and produce a well-structured llms.txt file. Pages are grouped by topic with concise descriptions suitable for LLM context windows.

How to use this tool

• Provide a URL — documentation sites, GitHub repos, API references, or any public website
• Configure scan depth — shallow for quick overviews, deep for comprehensive coverage
• Choose a detail level — overview for high-level structure, detailed for extended descriptions
• Download or copy the link — get a hosted URL you can reference directly in prompts

Common use cases

• Creating AI context files for open source projects so contributors and users can query docs with LLMs
• Generating documentation summaries for LLM-powered tools, chatbots, and RAG pipelines
• Building llms.txt for API documentation to help AI coding assistants understand your SDK
• Producing structured overviews of large sites for use in system prompts