Parsing TDMRep and AI.txt: Purpose-Based Scraping Controls

A robots.txt file answers only one question, whether a crawler may fetch this URL. That single bit no longer matches what site owners want to say in 2026. They want to allow search indexing but refuse model training, or to reserve text and data mining, and license it on request.

TL;DR

Purpose-based scraping controls express why you may use a page, not only whether you may fetch it. Reading them is now a build step.

• robots.txt carries one bit per crawler, so it can't say "search yes, training no" on its own.
• TDMRep declares a text-and-data-mining reservation through an HTTP header, a <meta> tag, or a /.well-known/tdmrep.json file.
• ai.txt states AI-training permission per file type, and defaults to opted out, the opposite of TDMRep.
• The signals conflict and overlap, so a gate resolves them per purpose into allow, deny, or needs-license.
• In a scan of 42 sites, 74% blocked an AI crawler, but only 21% served any TDMRep signal at all, and the richer formats almost never appeared.

This is educational, not legal advice. The law here is still moving, so check your own case with a lawyer in your jurisdiction.

Why robots.txt only gives you one bit

The Robots Exclusion Protocol was built to keep crawlers out of certain paths. It matches a crawler's product token to a group of Allow and Disallow rules, and that's the whole vocabulary. There's no field for purpose, and the protocol predates that question by 3 decades.

Site owners answered anyway, with hand-maintained lists of crawler names. Reading a real robots.txt shows the workaround, and its limits:

# Which AI crawlers does a site block at the root? robots.txt as a purpose signal.
import requests
from protego import Protego

UA = {"User-Agent": "Mozilla/5.0 (compatible; controls-demo/1.0)"}
AI_AGENTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "anthropic-ai",
             "Claude-Web", "CCBot", "Google-Extended", "Bytespider", "PerplexityBot",
             "Applebot-Extended", "meta-externalagent", "Amazonbot", "cohere-ai"]

def ai_blocks(origin):
    text = requests.get(origin + "/robots.txt", headers=UA, timeout=20).text
    rp = Protego.parse(text)          # real grouping, Allow overrides, and the * fallback
    return sorted(a for a in AI_AGENTS if not rp.can_fetch(origin + "/", a))

for site in ["https://www.nytimes.com", "https://www.reuters.com"]:
    hits = ai_blocks(site)
    print(f"{site:26} blocks {len(hits):2} AI crawlers: {', '.join(hits) or 'none'}")

Output:

https://www.nytimes.com    blocks 13 AI crawlers: Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, GPTBot, Google-Extended, OAI-SearchBot, PerplexityBot, anthropic-ai, cohere-ai, meta-externalagent

https://www.reuters.com    blocks 11 AI crawlers: Amazonbot, Bytespider, CCBot, Claude-Web, ClaudeBot, GPTBot, Google-Extended, PerplexityBot, anthropic-ai, cohere-ai, meta-externalagent

These 2 publishers use opposite mechanisms. The New York Times names each AI crawler and denies it. Reuters instead allowlists 86 bots in its July 2026 file and denies the rest with one User-agent: *, so it denies 11 of these agents without naming any. It allows OAI-SearchBot and ChatGPT-User but denies GPTBot, and the Times denies all 3.

The allowlist leaks, though. Reuters names Applebot and never names Applebot-Extended, so Apple's training token falls back to the Applebot group, and my scan reads that token as allowed. Both parsers I tried send a longer product token to the shorter name that's in the file, so a suffixed name inherits whatever the parent got.

Neither file states the purpose directly. Reuters is spelling "search yes, training no" as a list of vendor names. Blocking GPTBot stops OpenAI's training crawler, but the same content still gets to models through Common Crawl unless you also block CCBot. You're approximating one intent with a list of per-vendor on-off switches, and any new crawler name defeats that list.

That approximation is the problem that purpose-based controls exist to fix.

What purpose-based control means, and why it is now enforceable

A purpose-based control separates the use from the fetch. It lets one page say that search engines may index it, retrieval systems may quote it, and training may not touch it. The set of purposes is small. The same few purposes appear across the newer formats, even where each format spells them differently and carries only some of them:

• search: index the page and link back to it
• ai-train: train or fine-tune a generative model, the purpose behind pulling a whole site's text for an LLM
• ai-input: feed a model at answer time, also called retrieval-augmented generation (RAG)
• tdm: text and data mining, the broadest of the four in EU law, and the purpose that a normal pipeline passes to the gate

Most scraping work is none of the first three. A recital of the EU copyright directive says that private entities use text and data mining for complex business decisions, so price monitoring and market research count as tdm. Use ai-train or ai-input only when a model touches what you pulled.

For years a scraper could treat these signals as optional. Two EU rules changed that, and the second one carries fines from 2026.

The first is the Copyright in the Digital Single Market Directive (the DSM Directive). Its Article 4 lets anyone mine lawfully accessed content unless the rightsholder reserves that right "in an appropriate manner, such as machine-readable means". Silence means permission. A machine-readable reservation removes it.

The Times puts its Article 4 reservation in a comment. A comment is prose, and no parser reads prose. The scanner in the first section reads this same file, and it sees only the crawler rules below the comment. So the reservation is there for a human, but my scanner never reads it.

The second is the EU AI Act. Its Article 53 makes every general-purpose AI (GPAI) provider respect those Article 4 reservations through a copyright policy. The GPAI Code of Practice, published in July 2025 and signed by 23 providers as of July 2026 (OpenAI, Google, Anthropic and Microsoft among them), commits its signatories to crawl with robots.txt-aware crawlers and to honor machine-readable reservations. From 2 August 2026, breaching these duties risks a fine of up to 3% of global turnover or €15 million, whichever is higher.

These files declare a preference. They don't enforce it. In December 2025 a US court dismissed a claim that ignoring robots.txt is illegal circumvention. It ruled that such a file controls access no more than a sign telling visitors to "keep off the grass".

The force comes from copyright law and the AI Act, so honoring the file is a choice you make for compliance and reputation.

The signal landscape in 2026

Several formats now compete to carry that intent. They overlap, and they cover different points between "access control" and "licensing offer". Here's the map before the parsing:

Content-Signal is the one that lives in a file you already fetch. The Atlantic serves Content-signal: search=yes, ai-input=no, ai-train=no, which reads as "index me, do not quote me at answer time, and do not train on me". Here it is live:

Content-Signal is a robots.txt directive, so it belongs to a user-agent group like any other rule. The Atlantic gives that line to 4 named crawlers: BingBot, DuckAssistBot, FacebookBot and Googlebot. The * group carries none. If you read the file as one site-wide signal, you get an answer that was never written for your crawler.

Cloudflare's own robots.txt does the same thing. Its Content-Signal comes after the last named group, so strict grouping attaches it to Cohere-ai rather than to *.

Two things stand out. The purpose vocabulary isn't standardized. Cloudflare's Content Signals write ai-train=no, while the IETF's draft writes train-ai=n, for the same idea.

The vocabulary is also growing. In July 2026 Cloudflare was testing a fourth use field, with values immediate, reference and full, not yes or no at all. I found it live on Patreon, whose * group reads Content-Signal: search=yes,ai-train=no,use=reference. That line is inside a Cloudflare-managed block, so it's a default, not one site's choice.

The second thing is the spread. The formats span a spectrum, from "do not fetch" at the robots.txt end to "here is the price" at the licensing end. A parser has to read across all of it, one format at a time.

Parsing TDMRep

The TDM Reservation Protocol (TDMRep) is the format built for the DSM Directive's Article 4. It's a W3C Community Group report, finalized in May 2024. It's not a formal web standard, but it's the reservation format that real publishers deploy. It carries 2 properties: tdm-reservation (1 means reserved, 0 means not reserved) and an optional tdm-policy URL pointing to licensing terms.

A site can express those 2 properties in 3 ways for a web page, and those 3 ways form a precedence chain. The lowest priority is a /.well-known/tdmrep.json file that maps URL paths to reservations. An HTTP tdm-reservation header overrides that file for a given resource, and an HTML <meta name="tdm-reservation"> tag overrides both.

The spec puts EPUB and PDF metadata above even that, which matters if you mine the documents rather than the landing pages. A resource-level signal overrides a site-level one, so a parser that reads only the well-known file misses per-page overrides. A missing value at any level doesn't reset the one below it.

Three rules decide whether your reader is correct. Path matching in the well-known file is first-match-wins in document order, not longest-match. Only 0 and 1 are valid, and anything else falls back to unset, which means not reserved, so a malformed reservation silently becomes permission.

The third rule isn't about values but about the file. The spec says a JSON array, with brackets mandatory even for one rule, but in July 2026 Cambridge University Press served a bare object. If you loop over that, Python walks the keys, not the rules, so you lose a real reservation with its policy link. This reader coerces a lone object into a list, skips anything that's not a dict, and honors all 3:

# TDMRep resolver. Precedence low->high: well-known file < HTTP header < HTML meta.
# Reference: W3C TDMRep Community Group Final Report, 10 May 2024.
import json, re
from urllib.parse import urlsplit, urljoin
import requests
from bs4 import BeautifulSoup

UA = {"User-Agent": "Mozilla/5.0 (compatible; tdm-checker/1.0)"}

def _valid(v):                               # only 0 or 1 are valid, anything else is unset
    try: v = int(str(v).strip())
    except (TypeError, ValueError): return None
    return v if v in (0, 1) else None

def _loc_to_regex(location):                 # '*' = any run of chars, '$' = end of path
    out = ["^"]
    for i, c in enumerate(location):
        out.append(".*" if c == "*" else "$" if (c == "$" and i == len(location) - 1)
                   else re.escape(c))
    return re.compile("".join(out))

def resolve_tdm(url, s):
    parts = urlsplit(url)
    origin, path = f"{parts.scheme}://{parts.netloc}", parts.path or "/"
    reserved, policy, source = None, None, "none"
    try:                                     # 1. site-wide baseline from the well-known file
        wk = s.get(urljoin(origin, "/.well-known/tdmrep.json"), headers=UA, timeout=20)
        if wk.status_code == 200:
            doc = json.loads(wk.text)
            if isinstance(doc, dict):                # spec says array, some publishers ship one object
                doc = [doc]
            for rule in doc:                         # first matching rule wins, in order
                if not isinstance(rule, dict):
                    continue
                if isinstance(rule.get("location"), str) and _loc_to_regex(rule["location"]).match(path):
                    reserved, policy, source = _valid(rule.get("tdm-reservation")), rule.get("tdm-policy"), "well-known"
                    break
    except (requests.RequestException, json.JSONDecodeError, TypeError):
        pass                                 # a broken file is not a reservation
    try:                                     # 2. header, then meta, override the baseline
        r = s.get(url, headers=UA, timeout=20)
        if (h := _valid(r.headers.get("tdm-reservation"))) is not None:
            reserved, source = h, "http-header"
            policy = r.headers.get("tdm-policy", policy)
        if "html" in r.headers.get("content-type", ""):
            soup = BeautifulSoup(r.text, "html.parser")
            tag = soup.find("meta", attrs={"name": "tdm-reservation"})
            if tag and (m := _valid(tag.get("content"))) is not None:
                reserved, source = m, "html-meta"
            pol = soup.find("meta", attrs={"name": "tdm-policy"})   # the spec names a second meta tag
            if pol and pol.get("content"):
                policy = pol["content"]
    except requests.RequestException:
        pass
    return {"reserved": reserved, "policy": policy, "source": source}

if __name__ == "__main__":
    sess = requests.Session()
    for t in ["https://www.elsevier.com/", "https://www.lefigaro.fr/", "https://www.edrlab.org/",
              "https://www.nature.com/", "https://www.mondadori.it/", "https://example.com/"]:
        out = resolve_tdm(t, sess)
        label = {1: "RESERVED", 0: "not reserved", None: "unset (=not reserved)"}[out["reserved"]]
        print(f"{t:30} {label:22} via {out['source']:11} policy={out['policy']}")

Output:

https://www.elsevier.com/      RESERVED               via http-header policy=https://www.elsevier.com/tdm/tdmrep-policy.json
https://www.lefigaro.fr/       RESERVED               via well-known  policy=None
https://www.edrlab.org/        not reserved           via well-known  policy=None
https://www.nature.com/        RESERVED               via well-known  policy=https://dev.springernature.com/tdm/SNTDMPolicy.json
https://www.mondadori.it/      RESERVED               via html-meta   policy=None
https://example.com/           unset (=not reserved)  via none        policy=None

All three are live. Elsevier serves the header, Le Figaro serves the well-known file, Mondadori serves the meta tag, and my reader reports which one it read. Mondadori publishes both a well-known file and a meta tag, and the resolver reports html-meta, proving the precedence chain runs. The edrlab.org case is the rare live 0 (an explicit opt-in), and example.com is unset, which the law reads as permission.

Nature's file carries one rule, tdm-reservation: 1 for the whole site, with a tdm-policy link to Springer Nature's terms:

When a tdm-policy is present, it points to an ODRL (Open Digital Rights Language) document that states what mining is allowed, and on what terms. A reservation with a policy isn't a flat no. It means you must ask first, and the policy says whom to ask and under which conditions.

For each tdm:mine permission, pull the purpose constraint and any duty, and accept the prefixed odrl: operands that live files use:

# Reading the ODRL policy that a tdm-policy URL points to: per purpose, which duty applies.
import requests

UA = {"User-Agent": "Mozilla/5.0 (compatible; tdm-policy/1.0)"}

def _operand(node, side):                    # accept bare or odrl:-prefixed keys
    return node.get(side) or node.get(f"odrl:{side}")

def read_tdm_policy(policy_url, s):
    doc = s.get(policy_url, headers=UA, timeout=20).json()
    perms = doc.get("permission", [])
    perms = perms if isinstance(perms, list) else [perms]
    rules = []
    for p in perms:
        if "mine" not in str(p.get("action", "")):
            continue
        purpose = "any purpose"
        for c in (p.get("constraint") or []):
            if _operand(c, "leftOperand") == "purpose":
                purpose = f'{c.get("operator")} {_operand(c, "rightOperand")}'
        duties = [d.get("action") for d in (p.get("duty") or [])] or ["no duty"]
        rules.append((purpose, duties))
    contact = doc.get("assigner", {}).get("vcard:hasEmail", "")
    return {"contact": contact, "rules": rules}

if __name__ == "__main__":
    sess = requests.Session()
    for url in ["https://www.elsevier.com/tdm/tdmrep-policy.json",
                "https://dev.springernature.com/tdm/SNTDMPolicy.json"]:
        pol = read_tdm_policy(url, sess)
        print(f"\n{url}\n   contact: {pol['contact']}")
        for purpose, duties in pol["rules"]:
            print(f"   mine where purpose {purpose:28} -> {', '.join(duties)}")

Output:

https://www.elsevier.com/tdm/tdmrep-policy.json
   contact: mailto:tdm-license@elsevier.com
   mine where purpose eq stm:eu-dsm-article3       -> no duty
   mine where purpose neq stm:eu-dsm-article3      -> obtainConsent

https://dev.springernature.com/tdm/SNTDMPolicy.json
   contact: mailto:Datasolutions@springernature.com
   mine where purpose eq stm:eu-dsm-article3       -> obtainConsent
   mine where purpose neq stm:eu-dsm-article3      -> obtainConsent

Two publishers use the same standard and answer differently. Elsevier allows research mining with no duty, and requires consent for every other purpose. Springer asks for consent even for research, and each policy carries the email to ask. Parse the operands leniently, because the deployed files prefix them and use a purpose value (stm:eu-dsm-article3) that the base spec never defined.

Parsing ai.txt

ai.txt, proposed by Spawning, approaches the same problem from the media side. It lives at /ai.txt and says which file types a site will let you train on. Its model differs from TDMRep in 2 ways that matter to a parser.

It scopes by file type rather than by URL path, with the types grouped into 5 media categories: images, audio, video, text, and code. And it defaults to opted out: with no file, or with every category switched off, nothing may be trained on. In TDMRep silence means permission, but in ai.txt silence means refusal.

Take the syntax from Spawning's generator. The descriptions I found show a Disallow: ai_training: images form that the generator never emits. The generator writes robots.txt-style directives whose units are file extensions:

# ai.txt (Spawning): AI-training permission per file type. Default is opt-out.
# Syntax taken from Spawning's own generator output.
import re
from pathlib import PurePosixPath

def parse_ai_txt(text):
    rules = {}                                    # "*.jpg" / "/" / "*" -> allow | deny
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        m = re.match(r"(allow|disallow)\s*:\s*(\*\.\w+|\*|/)\s*$", line, re.I)
        if m:
            rules[m.group(2).lower()] = "allow" if m.group(1).lower() == "allow" else "deny"
    return rules

def may_train_on(rules, name):
    ext = PurePosixPath(name).suffix.lower()
    if (key := f"*{ext}") in rules:                # most specific: this file type
        return rules[key] == "allow"
    for site_wide in ("/", "*"):                   # then the site-wide line
        if site_wide in rules:
            return rules[site_wide] == "allow"
    return False                                   # Spawning's default: opted out

sample = """# Spawning AI
# Prevent datasets from using the following file types

User-Agent: *
Disallow: *.jpg
Disallow: *.mp4
Allow: *.txt
Allow: *.pdf
Allow: /
"""

rules = parse_ai_txt(sample)
for f in ["photo.jpg", "clip.mp4", "notes.txt", "paper.pdf"]:
    print(f"{f:11} -> {'allow' if may_train_on(rules, f) else 'deny'}")

Output:

photo.jpg   -> deny
clip.mp4    -> deny
notes.txt   -> allow
paper.pdf   -> allow

The parser is short because the format is short. The catch is adoption. The 42-domain scan later in this post finds no ai.txt at all, so in July 2026 I ran a second pass over 60 sites picked for this format. Not one served a valid file, and 46 answered with a 404.

Those 60 were the artist platforms and stock libraries that the format was written for, and Spawning's own domains served none. That list isn't in the companion script. Only the 42-domain scan is. "ai.txt" also names several unrelated proposals, an IETF draft and an academic language among them, so the name alone doesn't tell you which grammar you're reading.

Spawning's version is tied to the "Do Not Train" registry and has the generator behind it, but robots.txt AI tokens now carry most of the training opt-outs. Parse ai.txt if you meet one, because it's cheap. Don't expect it to carry your decision.

Parsing RSL: when the answer is a price, not a no

A layer above every opt-out format turns "no" into "not for free". Really Simple Licensing (RSL) is that layer, and it has live files behind it. It extends robots.txt with a License: line pointing to an XML file, and that file states, per purpose, what it permits, what it prohibits, and at what price. This reader fetches and normalizes it:

# RSL (Really Simple Licensing) reader. Discovery via the robots.txt `License:` line.
import re
import xml.etree.ElementTree as ET
from urllib.parse import urlsplit, urljoin
import requests

UA = {"User-Agent": "Mozilla/5.0 (compatible; rsl-reader/1.0)"}
NS = "{https://rslstandard.org/rsl}"

def find_license_url(origin, s, agent="*"):
    try:
        text = s.get(urljoin(origin, "/robots.txt"), headers=UA, timeout=20).text
    except requests.RequestException:
        return urljoin(origin, "/license.xml")
    groups, ags, lic, seen = [], [], None, False   # a License: line is scoped to its group
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        name, _, value = line.partition(":")
        name, value = name.strip().lower(), value.strip()
        if name == "user-agent":
            if seen:                               # a rule line closed the previous group
                groups.append((ags, lic)); ags, lic, seen = [], None, False
            ags.append(value.lower())
        else:
            seen = True
            if name == "license" and not lic:
                lic = value
    groups.append((ags, lic))
    for want in (agent.lower(), "*"):              # your group first, then the `*` group
        if hit := next((l for a, l in groups if want in a and l), None):
            return hit
    return urljoin(origin, "/license.xml")         # not a spec location, but where both live files sit

def parse_rsl(xml_text):
    root, out = ET.fromstring(xml_text), []
    for content in root.iter(f"{NS}content"):
        for lic in content.findall(f"{NS}license"):
            pay = lic.find(f"{NS}payment")
            out.append({
                "permits": [t for p in lic.findall(f"{NS}permits") for t in (p.text or "").split()],
                "prohibits": [t for p in lic.findall(f"{NS}prohibits") for t in (p.text or "").split()],
                "payment": pay.get("type") if pay is not None else None,
            })
    return out

def rsl_for(url, s):
    parts = urlsplit(url)
    lic_url = find_license_url(f"{parts.scheme}://{parts.netloc}", s)
    try:
        r = s.get(lic_url, headers=UA, timeout=20)
        if r.status_code == 200 and "<rsl" in r.text:      # trust the body, not the content-type
            return {"license_url": lic_url, "licenses": parse_rsl(r.text)}
    except (requests.RequestException, ET.ParseError):
        pass
    return {"license_url": lic_url, "licenses": []}

if __name__ == "__main__":
    sess = requests.Session()
    for site in ["https://www.theguardian.com/", "https://medium.com/"]:
        res = rsl_for(site, sess)
        print(f"\n{site}  ({res['license_url']})")
        for lic in res["licenses"]:
            print(f"   permits{lic['permits'] or ['-']} prohibits{lic['prohibits'] or ['-']} pay={lic['payment']}")

Output:

https://www.theguardian.com/  (https://theguardian.com/license.xml)
   permits['ai-train', 'ai-input'] prohibits['all'] pay=subscription

https://medium.com/  (https://medium.com/license.xml)
   permits['ai-input', 'ai-index', 'search'] prohibits['-'] pay=attribution
   permits['-'] prohibits['ai-train'] pay=subscription

Medium's file shows the whole argument in one document. It permits search, indexing, and retrieval under attribution for free, and prohibits training under a subscription payment that carries a form to ask. Medium's comment in the same file calls that an instruction to contact them so negotiation can begin. Search and training get different answers on the same content, which one robots.txt line cannot say.

The Guardian is stricter. It permits ai-train and ai-input under subscription, and prohibits every other use. So search is refused, not sold. My reader found both files through a License: line, and both were live in July 2026.

The RSL spec advertises its license files as application/rsl+xml in the Link header, but Medium serves plain application/xml. If you trust the advertised type you find nothing, so the parser checks the body for an <rsl root instead.

Combining signals into one purpose-based decision

Some sites publish more than one signal, and the signals can disagree. In my scan 19% carry 2 or more: robots.txt AI blocks plus a TDMRep reservation, or robots.txt AI blocks plus an RSL license.

One more format matters here, even at zero adoption. The IETF's AI Preferences working group (AIPREF) is standardizing a Content-Usage field, carried as an HTTP header or a robots.txt rule, with 2 categories (train-ai, search) and values y or n. In July 2026 it was a draft, not a published standard, and the working group was targeting August 2026 to send it to the IESG. I found no live deployment in my scan, but its resolution rule is the part I reused.

The job of a purpose gate is to take a URL and a purpose, read every signal, and return one verdict: allow, deny, or needs-license. When signals combine, any n wins, then any y, otherwise unknown. I add one rule to that. A paid license path outranks a plain block, because a License: line means the block is negotiable.

This gate combines the readers above, plus one that only ships in the companion: robots_signals. It returns the AI-crawler blocks per purpose, and the Content-Signal for the * group, which is the group a generic crawler falls into. Each signal contributes a per-purpose vote, and the votes resolve most-restrictive-first, except that an offered license comes back as needs-license:

# Purpose gate: read every signal for a URL, resolve one verdict per purpose.
# Reuses resolve_tdm and rsl_for from above. robots_signals ships in the companion:
# it parses the robots.txt groups and maps each AI token to a purpose.

def rsl_state(rsl, purpose):
    want = {"ai-train": {"ai-train", "ai-all", "all"}, "ai-input": {"ai-input", "ai-all", "all"},
            "search": {"search", "all"}, "tdm": {"ai-train", "ai-all", "all"}}[purpose]
    rank = {"all": 0, "ai-all": 1}                   # `all` is the broadest token, the purpose token the narrowest
    def hit(tokens):                                 # how specific is the rule that covers this purpose
        return max((rank.get(t, 2) for t in want & set(tokens)), default=-1)
    allow = needs = deny = False
    for lic in rsl["licenses"]:
        commercial = lic["payment"] not in (None, "free", "attribution")
        p, q = hit(lic["permits"]), hit(lic["prohibits"])
        if p > q:                                    # the more specific rule permits it
            needs = needs or commercial              # permitted, but only if you pay
            allow = allow or not commercial          # permitted for free / attribution
        elif lic["permits"]:                         # the permits list is closed and this use is not on it
            deny = True                              # the payment prices the permits, not this
        elif q >= 0:                                 # prohibited, and no permits list to price
            needs = needs or commercial              # prohibited by default, but licensable
            deny = deny or not commercial            # prohibited outright, no path offered
    return "needs-license" if needs else "deny" if deny else "allow" if allow else None

def evaluate(sig, purpose):
    votes = []
    if purpose in ("tdm", "ai-train") and sig["tdm"]["reserved"] == 1:      # TDMRep reservation
        votes.append(("TDMRep", "needs-license" if sig["tdm"]["policy"] else "deny"))
    for tok, pur in sig["robots"]["ai_blocks"].items():                     # robots.txt AI blocks
        if pur == purpose:
            votes.append((f"robots:{tok}", "deny"))
    cs = sig["robots"]["content_signal"].get(purpose)                       # Cloudflare Content-Signal
    if cs in ("yes", "no"):
        votes.append(("Content-Signal", "allow" if cs == "yes" else "deny"))
    if (rs := rsl_state(sig["rsl"], purpose)):                              # RSL license
        votes.append(("RSL", rs))
    return votes

def resolve(votes):
    states = [s for _, s in votes]
    return ("NEEDS-LICENSE" if "needs-license" in states else "DENY" if "deny" in states else "ALLOW" if "allow" in states else "UNKNOWN")

Run against the live sites, the gate turns those signals into decisions a pipeline can branch on:

site               search         ai-train        ai-input        tdm
elsevier.com       UNKNOWN        NEEDS-LICENSE   UNKNOWN         NEEDS-LICENSE
theguardian.com    DENY           NEEDS-LICENSE   NEEDS-LICENSE   NEEDS-LICENSE
medium.com         ALLOW          NEEDS-LICENSE   ALLOW           NEEDS-LICENSE
stackoverflow.com  DENY           DENY            DENY            UNKNOWN
nature.com         UNKNOWN        NEEDS-LICENSE   DENY            NEEDS-LICENSE

The verdicts read the way the sites intend. Medium allows search and retrieval, and makes training negotiable. Nature licenses training but denies real-time retrieval, and it blocks the RAG crawlers with no license path.

Stack Overflow denies search and training in its Content-Signal, and its blanket Disallow: / catches retrieval too. The Guardian prices training and retrieval, and refuses the rest. None of that fits a yes-or-no flag, so the gate keeps each reason attached for a human to audit.

That is the resolution logic, and it is not deployable yet. Pull robots_signals from the companion to run it. The pipeline section at the end adds the caching and the default it still needs.

What sites actually publish

The gate has nothing to read unless sites publish signals, so I measured what they publish. In July 2026 I scanned 42 domains and recorded which signals each one serves. The companion script is on GitHub, so you can reproduce the numbers.

The list covers academic publishers, European and US news, the RSL backers, reference and developer sites, and ecommerce. It also includes edrlab.org and cloudflare.com, the 2 authors of TDMRep and Content-Signal, because how an author uses its own standard is worth knowing.

The scan runs on plain requests. If a site times out, or answers a bare client with a challenge page, it stays in the denominator and counts as publishing nothing. So read the numbers below as minimums, and a re-run can move one either way.

scanned 42 domains, 42 usable
robots.txt blocks an AI crawler : 31/42 = 74%
TDMRep signal (any method)      :  9/42 = 21%
RSL License                     :  2/42 = 5%
Cloudflare Content-Signal       :  1/42 = 2%
AIPREF Content-Usage            :  0/42 = 0%
ai.txt                          :  0/42 = 0%
two or more signal types        :  8/42 = 19%
conflicting ai-train verdicts   :  3/42 = 7%

Here's the bar chart of what 42 sites publish:

robots.txt AI blocking is mainstream, and everything purpose-based is early. The one Content-Signal is Stack Overflow. Cloudflare serves one too, but it belongs to the Cohere-ai group, so my scan reads the * group where a generic crawler looks, and counts one.

The TDMRep signals came almost entirely from academic and European publishers, the sites that the standard was written for. Eight of the nine carry a reservation. The ninth is edrlab.org, the lab that wrote the protocol, and it publishes an explicit 0 to say mining is fine.

That European skew shows up outside my sample too. EDRLab reported that 143 of the top 250 French sites had implemented TDMRep as of 2024. That's more than half the top sites in a single country, against the 21% my global scan found across a mixed sample.

Of the 42, 3 carried a training verdict that was deny from robots.txt and needs-license from a TDM policy or RSL file at once. robots.txt blocks the use, and the license offers to sell it. A gate that reads only one of them gets the wrong answer.

Independent studies read the same way. The Data Provenance Initiative's Consent in Crisis measured this early: in a single year, 2023 to 2024, 28%+ of the most actively maintained sources in C4 went fully restricted through robots.txt. And Cloudflare measured 52% of crawler requests going to AI training by June 2026, up from 22% a year earlier.

So more sites block AI crawlers every year, but almost none of them say why in a way my scan can read. I get one bit, and I still have to guess the purpose behind it.

The limits of these signals

Reading these files well still leaves 3 gaps to design around.

The first is who carries the burden. Silence means permission under TDMRep and the DSM Directive, so a rightsholder who publishes nothing has reserved nothing, and most publish nothing. You cannot fix that from the scraper side: you read what is there, you record what you read, and your default decides the rest. Whether you may fetch a page at all is a separate question, with its own case law.

The second is fragmentation. A single purpose, "do not train", can arrive as a robots.txt rule, a TDMRep reservation, an ai.txt line, a Content-Signal, or a Content-Usage header, and 2 of those spell the category differently. The European Commission ran a consultation into early 2026 on which machine-readable opt-out to endorse, and plans to publish an agreed list. Until that list arrives, a compliant reader supports several formats, not one.

The third is the standard for "machine-readable" itself. A German court held in December 2025 that a reservation counts only if a machine can both perceive and interpret it, not merely if a person can read it. No EU-wide answer exists yet, so favor a structured signal like TDMRep over buried prose, and record which signal you read and when.

The EU's Court of Justice may change all of this. It heard its first generative-AI copyright case in March 2026. One question that was referred to it asks whether training a model falls under the Article 4 exception at all.

Building purpose-awareness into your pipeline

You still have to read these signals, so the gate has to work at scale. The naive version does not work at scale. It fetches 3 files per URL before it even looks at the page: robots.txt, the well-known reservation, and the license. A million URLs means millions of extra requests, and almost all are redundant.

ai.txt does not vote here, because it states permission per file type and cannot answer a URL-level question.

Those 3 files are per-origin, not per-page, so fetch them once per host and cache them. But cache the well-known file's rules, not its answer, because it matches paths and every URL needs its own match. The tdm-reservation header and <meta> tag are per-page, and they arrive on the response you already fetched, so they cost nothing extra. That split makes the gate deployable:

# Production gate. It caches the origin signals per host, reads the page reservation off
# the response you already fetched, and resolves an UNKNOWN with a default you choose.
# It builds on the readers above. robots_signals ships in the companion.
import time, json, threading
from urllib.parse import urlsplit, urljoin
from bs4 import BeautifulSoup

def tdm_rules(origin, s):                        # the file is per-origin, so cache this part
    try:
        r = s.get(urljoin(origin, "/.well-known/tdmrep.json"), headers=UA, timeout=20)
        doc = json.loads(r.text) if r.status_code == 200 else []
    except (requests.RequestException, json.JSONDecodeError):
        return []                                # a broken file is not a reservation
    if isinstance(doc, dict):                    # some publishers ship one object
        doc = [doc]
    return doc if isinstance(doc, list) else []

def tdm_match(rules, path):                      # the match is per-URL, so do not cache this part
    for rule in rules:                           # first matching rule wins, in order
        if not isinstance(rule, dict):
            continue
        if isinstance(rule.get("location"), str) and _loc_to_regex(rule["location"]).match(path):
            return _valid(rule.get("tdm-reservation")), rule.get("tdm-policy")
    return None, None

def page_reservation(resp):                       # works on a plain fetch or a scraping-API response
    h = resp.headers                              # a proxy API may prefix origin headers (ScrapingBee: spb-)
    reserved = _valid(h.get("tdm-reservation") or h.get("spb-tdm-reservation"))
    policy = h.get("tdm-policy") or h.get("spb-tdm-policy")
    if "html" in (h.get("content-type") or h.get("spb-content-type") or ""):
        tag = BeautifulSoup(resp.text, "html.parser").find("meta", {"name": "tdm-reservation"})
        if tag and _valid(tag.get("content")) is not None:
            reserved = _valid(tag.get("content"))
    return reserved, policy

_cache, TTL = {}, 86400
_locks = [threading.Lock() for _ in range(64)]    # striped: the lock table stays bounded

def _origin_signals(origin, s):                       # fetched once per host, then cached
    if origin not in _cache or time.time() - _cache[origin][0] > TTL:
        with _locks[hash(origin) % len(_locks)]:      # one fetch per cold origin, not one per worker
            if origin not in _cache or time.time() - _cache[origin][0] > TTL:
                _cache[origin] = (time.time(), {"robots": robots_signals(origin, s), "rsl": rsl_for(origin, s),
                                  "tdm_rules": tdm_rules(origin, s)})
    return _cache[origin][1]

def _decide(url, purpose, tdm, o, default):
    votes = evaluate({"tdm": tdm, "robots": o["robots"], "rsl": o["rsl"]}, purpose)
    verdict = resolve(votes)
    verdict = verdict if verdict != "UNKNOWN" else default
    return verdict, {"url": url, "purpose": purpose, "verdict": verdict,
                     "signals": [n for n, _ in votes] or ["none"], "checked_at": int(time.time())}

def origin_verdict(url, purpose, s, default="ALLOW"):   # ask before you fetch, and skip what is already denied
    parts = urlsplit(url)
    o = _origin_signals(f"{parts.scheme}://{parts.netloc}", s)
    wr, wp = tdm_match(o["tdm_rules"], parts.path or "/")
    return _decide(url, purpose, {"reserved": wr, "policy": wp}, o, default)

def gate(url, purpose, resp, s, default="ALLOW"):     # ALLOW = fail-open, DENY = fail-closed
    parts = urlsplit(url)
    o = _origin_signals(f"{parts.scheme}://{parts.netloc}", s)
    wr, wp = tdm_match(o["tdm_rules"], parts.path or "/")   # every URL needs its own match
    pr, pp = page_reservation(resp)                        # page-level overrides origin-level
    return _decide(url, purpose, {"reserved": pr if pr is not None else wr, "policy": pp or wp}, o, default)

Running it over 3 URLs on one host, then the lineage record and a no-signal host:

https://www.nature.com/           ai-train -> NEEDS-LICENSE
https://www.nature.com/nature/    ai-train -> NEEDS-LICENSE
https://www.nature.com/subjects/  ai-train -> NEEDS-LICENSE
origins fetched: 1  (not 3)
lineage: {"url": "https://www.elsevier.com/", "purpose": "ai-train", "verdict": "NEEDS-LICENSE", "signals": ["TDMRep"], "checked_at": 1784027285}
no-signal host: fail-open -> ALLOW, fail-closed -> DENY

The gate forces two operational choices. The first is the default: with no signal, UNKNOWN becomes ALLOW (fail-open, matching silence-is-permission) or DENY (fail-closed). Apply the same default when a signal fetch fails, so a 503 on robots.txt does not silently flip you to permissive. The second is the record: that lineage line is what an auditor or a buyer asks for, so store it beside the row it gated.

One caveat applies to the readers. The token scan asks whether a fixed list of AI crawlers may fetch the root, and it is a survey, not a crawl-decision engine. For the real "may my bot fetch this path" question, run the same protego rules against your own product token and the exact URL.

Don't write your own robots.txt parsing either. A naive reader misses sites that allowlist named bots and deny the rest with one User-agent: *. That's how Reuters denies those 11 crawlers without naming any. And verified-bot schemes, where a crawler signs its requests, are starting to gate access, so a 2026 crawler must also prove who it is.

Putting it into practice

Most of what you meet is still a block with no stated purpose, as the scan showed. So the purpose you act on is usually one you inferred, from a crawler name or from silence, and the site never declared it.

You own the gate. The fetch under it is the part you can buy.

Reading the control files needs only plain requests. Fetching the content they govern means getting past anti-bot and rendering, and a scraping API like ScrapingBee does both.

It hands you the page reservation too. A fetch through it returns the origin's headers prefixed with spb-, so the reservation arrives as spb-tdm-reservation and the gate reads it off the same response.

You read the origin signals before the fetch, so origin_verdict can deny a URL before you spend a fetch on it. You read the page reservation after it, off the response you already have. The gate doesn't stop the fetch. It decides what you may do with the page once you have it.

Read the signal before you write the scraper. Get a free API key and put the fetch layer under a gate you own.

Frequently asked questions

Is TDMRep legally binding?

TDMRep is a format, not a law. Its force comes from EU rules: the DSM Directive removes the TDM exception once a machine-readable reservation exists, and the AI Act adds a duty on GPAI providers. But Article 4 covers commercial mining too, so a reservation removes your exception even if you train nothing.

What is the difference between robots.txt and TDMRep?

robots.txt controls fetching: whether a crawler, named or caught by the * group, may request a URL. TDMRep controls use: whether already-fetched content may be mined, and under what license. A page can be allowed in robots.txt yet reserved in TDMRep, so the two are complementary layers, not substitutes.

Does ai.txt actually stop AI training?

No file stops training on its own. Each one declares a preference. ai.txt states, per file type, that training isn't permitted, and a compliant crawler honors it. Adoption is low, and most sites express training opt-outs through robots.txt AI tokens instead, so ai.txt is one signal among several.

Is llms.txt an AI scraping control?

No. llms.txt tells a model which of your pages to read at answer time. It doesn't say who may use those pages, or for what, so a purpose gate has nothing to read there. For control, read robots.txt, TDMRep, Content-Signal, or RSL.

How do I handle conflicting AI-usage signals?

Resolve them per purpose, not per site. For each purpose, collect every signal's vote and apply a most-restrictive rule: a deny outweighs an allow, and a paid license path outranks a plain block. Keep the reasons attached, so a denied or licensable URL is auditable later.

Do I need to parse these signals if I scrape outside the EU?

Training outside the EU doesn't exempt you. The AI Act's copyright duty applies to any general-purpose model placed on the EU market, wherever it was trained. Beyond the law, a site can block a crawler that ignores its file, wherever that crawler runs.

What is the fastest way to check one URL right now?

Fetch 3 things: /robots.txt, /.well-known/tdmrep.json, and the page's own tdm-reservation header. Those 3 cover the most-deployed signals in my July 2026 scan. Add RSL and Content-Signal parsing as you meet sites that publish them.