Create your llms.txt

Use the llms.txt file to provide structured information about your knowledge base to AI language models. While robots.txt controls whether crawlers can access your content, llms.txt helps AI tools understand your content when they do access it, giving them a short map of what's available, what it's for, and where to find it.

Unlike robots.txt, KnowledgeOwl doesn't generate a default llms.txt file. Leaving the Custom llms.txt field blank means your knowledge base will not publish an llms.txt file at all. To publish one, you'll need to write and save the content yourself.

What is llms.txt?

llms.txt is an emerging standard that provides a plain-text, Markdown-formatted overview of a website for large language models (LLMs) to read. It's basically the robots.txt file for AI tools.

When an AI tool visits your knowledge base, it reads your llms.txt file to get a structured summary of your content, such as your knowledge base's purpose, key sections, and links to important pages, rather than having to crawl and interpret everything from scratch.

Your llms.txt file is publicly available at yourdomain.com/llms.txt once you save content to the field.

For more information on the llms.txt file, refer to https://llmstxt.org/.

How to add or customize llms.txt

To add or customize your llms.txt file:

1. Go to KB settings > Domain.
2. Enter your llms.txt content in the Custom llms.txt text box using Markdown format. Refer to the Quickstart below for examples.
3. Be sure to Save your changes.
4. Select the llms.txt link in the Custom llms.txt section to verify that your file looks correct:
   Select the llms.txt link to verify that your file looks correct

Quickstart: llms.txt format and examples

The llms.txt format uses Markdown. 

A basic llms.txt file typically includes:

• A top-level heading with your site or company name (line begins with # )
• A blockquote with a brief description or summary of your knowledge base (line begins with > )
• One or more heading 2 sections with links to key areas of your knowledge base (line begins with ## ).
  • Add each link on a new line using this format:

    - [Link label](https://help.acme.com/help/my-link): Description of what this page contains.

• An Optional heading 2 section containing links to secondary key areas, formatted just like the other sections and link lists. If an LLM tool needs shorter context, it will skip this section. This can be a great place to link to your full sitemap.

Refer to the Markdown Cheat Sheet for more information on formatting Markdown.

Basic llms.txt example

Here's what a sample knowledge base llms.txt file might look like:

# Acme Help Center

> Official documentation and support articles for Acme products.

## Getting Started
- [Quick Start Guide](https://help.acme.com/help/quick-start): Set up your account and get started fast.
- [System Requirements](https://help.acme.com/help/system-requirements): What you need to run Acme software.

## Account & Billing
- [Managing Your Subscription](https://help.acme.com/help/subscription): Upgrade, downgrade, or cancel.
- [Invoices and Payments](https://help.acme.com/help/invoices): Download invoices and update payment methods.

## Troubleshooting
- [Common Error Messages](https://help.acme.com/help/errors): What they mean and how to fix them.

## Optional
- [Full sitemap](https://help.acme.com/help/sitemap.xml)

For a general website example of an llms.txt file, refer to https://llmstxt.org/#example and https://www.fastht.ml/docs/llms.txt.

Tips for writing a useful llms.txt

Follow these tips to create an effective llms.txt file:

• Use concise, clear language. Avoid ambiguous terms or unexplained jargon.
• Keep descriptions short and informative. One sentence per link is usually enough. LLMs use these to decide which URLs to fetch for more detail.
• Prioritize your most important content. You don't need to list every article. Focus on top-level categories and your most commonly referenced pages.
• Use your actual public URLs. If your knowledge base is on a custom domain, use that domain in your links.
• Update it when your structure changes. If you add major new sections or reorganize categories, update your llms.txt to reflect those changes.

Learn more

For more background on the llms.txt standard:

• llmstxt.org: The original llms.txt specification and proposal
• LLMS Central's What is the llms.txt File? A Complete Guide
• LLMS Central's Complete Guide to AI Bot User-Agents: Broader context on how AI tools interact with the llms.txt file and websites