Settings – AI crawlers allowlist

Overview

The options in the AI crawlers allowlist panel let you control which AI crawlers can access your site’s content. By enabling allowlist mode, you can block all crawlers by default and explicitly authorize only selected trusted AI systems based on their User-Agent. This feature helps you prevent unauthorized or aggressive AI crawling, protect your data, and ensure your content is used only by AI platforms you approve.

Doc image optimizer 240 01

Available Options

Enable allowlist mode

  • Description: When enabled, all AI crawlers are blocked by default. Only those selected below will be allowed to access your site. This option is recommended for maximum control.

Authorized AI crawlers

  • Description: Choose the AI crawlers you want to authorize from the list (e.g., OpenAI GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, PerplexityBot, Amazonbot, CCBot, YouBot, bingbot, Bravebot).
    Each option shows the User-Agent string and a description of the crawler/platform.
    Only the selected crawlers will be able to scan your content according to your rules.