Settings – File content options

Overview

The options in the File content options panel allow you to enrich the llms.txt file with structured metadata, details, and directives that improve the understanding and scanning of the site by AI crawlers and LLM systems.

Doc image optimizer 240 01

Available Options

Include Disallow directives

  • Description: Adds “Disallow” directives to the llms.txt file to tell AI crawlers which areas of the site should not be crawled (e.g., cart, checkout, admin, etc.). Recommended for blocking sensitive routes and protecting privacy.

Include Allow directives

  • Description: Adds “Allow” directives to highlight priority and important paths on the site, making it easier for AI to scan relevant content.

Include Priority URLs section (YAML)

  • Description: Inserts a structured “Priority URLs” section in YAML format, with detailed metadata for each URL (title, date, type, author, etc.). Useful for advanced crawlers that leverage structured data.

Include publication dates in metadata

  • Description: When enabled, adds the publication date for each piece of content to the metadata.
    Allows LLMs to analyze the freshness and historicity of content.

Include post type in metadata

  • Description: Adds the post type (custom post type) to the YAML data, allowing crawlers to distinguish between standard and custom content (e.g., products, events, portfolios).

Include summary (from excerpt)

  • Description: It adds a summary taken from the excerpt to each metadata entry, providing a brief textual description that helps AI crawlers understand the essence of each piece of content.

Include tags/categories

  • Description: It adds tags and categories as arrays, enriching the semantics of the data and facilitating automatic content classification.

Include author

  • Description: Adds the author’s name to each piece of content, which is useful for correctly attributing the source and improving transparency.

Include language

  • Description: Adds the “language” field to metadata, detecting the language of the content (compatible with plugins such as Polylang or WPML, otherwise uses the language of the site).