# Olivares AI - robots policy
# -----------------------------------------------------------------------------
# Search & answer engines: fully welcome. We publish /llms.txt and /llms-full.txt
# so assistants can describe Olivares AI accurately and cite it well.
#
# AI *training* crawlers: only Anthropic (Claude) is allowed; every other model-
# training crawler is blocked. This is a deliberate, documented stance - search
# and answer/citation bots (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot,
# DuckAssistBot, ...) keep full access via the default group below.
#
# Legal pages (privacy / terms / imprint, every locale) stay live and
# footer-linked for legal compliance while being kept OUT of search AND out of
# AI training - by two DIFFERENT mechanisms, and the difference is the whole
# point of how this file is written:
#
#   - Out of SEARCH by the "noindex" they carry (LegalPage.astro). They are
#     deliberately NOT disallowed for search crawlers. A crawler that cannot
#     download the page cannot read its "noindex", so blocking it here is what
#     makes Google index the URL with no content - the exact opposite of the
#     intent. Measured 2026-08-06: 4 such URLs were indexed that way and the
#     series was still climbing (2 -> 3 -> 5).
#
#   - Out of AI TRAINING by the Disallow lines in the ClaudeBot group below.
#     A training crawler obeys robots.txt and does NOT read "noindex", so that
#     group is the only thing keeping these pages out of a corpus. It stays.
#
# So: for SEARCH crawlers nothing is disallowed here at all. The legal pages are
# disallowed only in the ClaudeBot group. (The blocked training crawlers further
# down carry "Disallow: /" - that is a different rule and a different purpose.)
#
# A user-agent with its own group below reads ONLY that group, so the per-bot
# rules override the catch-all.

# Default: everything crawlable (search, answer engines, user-initiated AI).
# The legal pages are crawlable here ON PURPOSE - their "noindex" is what keeps
# them out of the index, and it can only be read if the page can be fetched.
# Sources: support.claude.com art. 8896518 - 2026-04-07 (Claude-User /
# Claude-SearchBot); developers.openai.com/api/docs/bots (OAI-SearchBot /
# ChatGPT-User).
User-agent: *
Allow: /

# AI training - ALLOWED (Anthropic / Claude only), minus the legal pages.
# These six lines are NOT duplicated in the catch-all group above, and that
# asymmetry is deliberate - see the header. Removing them would put the legal
# pages into a training corpus; "noindex" would not stop that.
# Source: support.claude.com art. 8896518 - 2026-04-07.
User-agent: ClaudeBot
Allow: /
Disallow: /privacy
Disallow: /terms
Disallow: /imprint
Disallow: /*/privacy
Disallow: /*/terms
Disallow: /*/imprint

# Claude-User and Claude-SearchBot intentionally have no group here; they fall
# to the default User-agent: * Allow.

# AI training - BLOCKED (every model-training crawler except Anthropic).
# Source: developers.openai.com/api/docs/bots (GPTBot).
User-agent: GPTBot
Disallow: /

# Source: google-common-crawlers - 2026-04-23; Google-Extended does not affect
# Search or AI Overviews.
User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: PanguBot
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: Ai2Bot-Dolma
Disallow: /

# Discovery
Sitemap: https://olivares.ai/sitemap-index.xml