Custom robots.txt support
Publish a robots.txt file in your vault to control how search engines crawl your site.
You can now control search engine crawling by publishing a robots.txt file from your vault. If one is found in your content root, it takes precedence over the default auto-generated one.
This is useful for blocking specific crawlers (e.g. AI bots) or restricting certain paths from being indexed.
See Custom robots.txt docs for details.
