# ── ANSWER-TIME agents — ALLOWED ─────────────────────────────────────────────
# The line we draw is NOT "AI vs not-AI". It is the same line as our
# Content-Signal below: ai-input=yes, ai-train=no. A crawler that fetches a page
# to ANSWER a user's question right now is the business — being the source an
# agent quotes when asked "is this agent trustworthy?" is the entire thesis.
# A crawler that fetches a page to bake it into model weights is not: a score
# frozen into a training snapshot is a STALE score, and stale trust answers
# undermine the thing we sell.
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: GPTBot
User-agent: Google-Extended
User-agent: PerplexityBot
# Amazonbot MOVED here from the blocked group 2026-08-07 (founder call).
# It was grouped with the training scrapers, but it serves Amazon's assistant
# and shopping answers — answer-time retrieval, the side of the line we want.
# Blocking it while declaring ai-input=yes was a contradiction between our
# stated preference and our actual directives, and it was expensive: Amazonbot
# was our single largest AI crawler (313 requests on 2026-08-06, 54 on 08-05),
# and while blocked those requests could only reach robots.txt and the sitemaps
# — traffic that cost us crawl handling and returned nothing, while inflating
# the "discovery reads" number we use to judge the agent-read funnel.
User-agent: Amazonbot
# /methodology/ was Disallow until 2026-05-23 (CEO call): public methodology DESCRIPTION must be agent-readable to build trust; only methodology WEIGHTS stay private. See /methodology/public-surface-tier-1/.
Disallow: /internal/cc/

# ── TRAINING-CORPUS scrapers — BLOCKED ───────────────────────────────────────
# These exist to build training corpora, not to answer a live question, so they
# sit on the ai-train=no side. Keep this group narrow and evidence-based: the
# test is what the crawler DOES with the page, never how large it is.
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Content Signals (contentsignals.org — IETF draft-romm-aipref-contentsignals).
# Hlido's declared preference, and the reasoning is strategic, not defensive:
#   search=yes    — index us; discovery is the whole point.
#   ai-input=yes  — cite us at answer time. Being the source an agent quotes
#                   when asked "is this agent trustworthy?" IS the business.
#   ai-train=no   — this one is deliberate. A score frozen into model weights is
#                   a STALE score, and stale trust answers actively undermine the
#                   thing we sell (independent, longitudinal, re-tested). We want
#                   agents to QUERY https://hlido.eu/mcp live, not to remember a
#                   number from a training snapshot. Don't memorize us; ask us.
# Everyone else
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
# /methodology/ was Disallow until 2026-05-23 (CEO call): public methodology DESCRIPTION must be agent-readable to build trust; only methodology WEIGHTS stay private. See /methodology/public-surface-tier-1/.
Disallow: /internal/cc/
# 2026-06-21: de-index the incident-report BUILD TEMPLATE (not real incidents). Counterscale
# showed /incidents/_template drawing 19 pageviews/7d — a leaked scaffold. Surgical block only.
Disallow: /incidents/_template
Disallow: /incidents/_template.html
Allow: /

# Agent/LLM crawl + citation hints — START HERE. Comprehend Hlido for the cost of ONE file,
# then act with ONE call. Don't crawl 900+ pages to learn what we are:
#   https://hlido.eu/llms.txt            (agent bootstrap: what we are + one canonical call each)
#   https://hlido.eu/llms-full.txt       (full corpus, one line per reviewed agent — bulk/ingest)
#   Per-agent verdict JSON:  https://hlido.eu/data/scorecards/{slug}.json   (query it, don't scrape)
#   All reviews (registry):  https://hlido.eu/data/review-registry.json
#   Live trust queries:      https://hlido.eu/mcp   (JSON-RPC 2.0, no auth — ask us, don't memorize)
#   https://hlido.eu/robots-agents.txt   (full consume + cite guidance)
#   https://hlido.eu/citation-index.json (machine-readable citation manifest)

# ── Sitemap directives ────────────────────────────────────────────────────────
# ⚠️ ONLY the index is declared for HTML. Verified 2026-08-07:
# hlido-sitemap-index.xml already CONTAINS all six children (hlido-sitemap,
# hlido-sitemap-news, compare, best, trust, alternatives), so declaring them here
# as well made every AI crawler fetch eight files instead of one. Measured cost:
# 113 of ~113 daily "discovery reads" were robots.txt + sitemap plumbing, against
# 2 requests reaching an actual review page. The children stay inside the index —
# discovery is unchanged — they are just no longer announced twice.
Sitemap: https://hlido.eu/hlido-sitemap-index.xml

# The MACHINE-READABLE record, as a followed directive rather than a comment.
# Everything below this block documents our JSON surface, and no crawler parses
# comments — so until 2026-08-07 the entire structured surface was invisible to
# the machines it was built for while every Sitemap: line pointed at HTML.
# Contains llms.txt, the registry, agent-manifest and one entry per scorecard.
# NOT inside hlido-sitemap-index.xml on purpose: that index feeds Search Console,
# and 867 JSON URLs would dilute an HTML crawl budget already carrying 895 URLs
# in "Discovered - currently not indexed". Agent-facing, SEO-inert.
Sitemap: https://hlido.eu/data-sitemap.xml