🤖 New: Customize your robots.txt file

Beep borp boop! The humans here at KnowledgeOwl are excited to tell you that you can now customize your robots.txt file.

This means you now have the ability to:

1. Explicitly allow certain bots or crawlers to crawl your public knowledge base.
   This is helpful if your public knowledge base is a part of your SEO/AEO/GEO strategy.

2. Explicitly prevent certain AI bots or crawlers from crawling your public knowledge base.
   This is helpful if you want to prevent AI companies from crawling your public knowledge base, and using your content to train their models.

The setting for this lives in KB settings > Domain.

There are also options to allow and disallow crawling based on different contexts (like training vs. searching). You can find copyable code samples in the Customize your robots.txt article based on different scenarios, so you can make these changes without having to think too much about it.

Have questions or feedback about the new robots.txt options? Contact us; we'd love to hear from you!