Overview
The options in the Excludes panel allow you to specify which paths and content should be blocked from AI/LLM crawling. This helps you protect sensitive or irrelevant areas of your site and prevents large language models from accessing content you do not want included in their training or indexing.
Available Options
Custom blocked paths
- Description: Enter one path per line to manually block specific folders or URLs from AI crawling.
These will be added to the default exclusions (such as cart, checkout, admin areas).
Each path must start with/(e.g.,/private/,/wp-content/uploads/2024/).
Useful for excluding confidential, temporary, or irrelevant sections of your website.
Auto-disallow no-index pages
- Description: When enabled, all pages and posts marked with the meta tag “noindex” will be automatically blocked for AI crawlers.
This is recommended if you use SEO plugins and want to ensure that LLMs do not crawl content explicitly set as “noindex”.
Compatible with popular SEO plugins: Yoast SEO, Rank Math, SEOPress, All in One SEO Pack, Slim SEO. The panel also shows a list of currently excluded “noindex” posts/pages for your reference.
