← Home

Methodology

This is the full, public version of the rubric Interrupt Index scores against — every number below is imported directly from the same source code the scoring engine runs, so this page can’t quietly drift out of sync with real scores.

The headline: Overall LLM-Friendliness (Composite)

The big number on every tool page, in the directory, and on the category leaderboards is the Overall LLM-Friendliness score — a composite of the two things that decide whether an agent can actually use a tool: how much it forces a human into the loop (the Interrupt Score) and how machine-legible it is (the Machine-Readability Score). It is a flat, uniform 50/50 mean:

Overall = round( (Interrupt Score + Machine-Readability Score) / 2 )

The split is a deliberate flat mean for every listing — the same weighting whether a tool is one your agent runs on or one it visits — rather than a per-usage-mode weighting, so the headline stays a single, comparable number that nobody can accuse of being tuned per category. When either input hasn’t been measured yet, there is no composite and the listing reads “not yet scored.”

What counts as an interrupt

An interrupt is any point in a tool’s flow where a human must act before an otherwise-autonomous agent can proceed — a CAPTCHA, an email/SMS code, a “contact sales” wall, a terms-of-service clause banning bots, a forced redirect to finish somewhere else. Every finding our probes record is tagged along four independent axes:

Two things are deliberately kept out of this score and reported as separate companion signals instead — machine-readability quality and custody/risk posture. See Companion signals below for why, and how each is computed.

The interrupt taxonomy

Base weight is a 0–10 scale for how disruptive one instance of that interrupt type is, independent of how often it recurs — recurrence is handled separately by the timing multiplier below. The “typical” timing and hardness columns describe common real-world patterns; the actual timing/hardness used in scoring always comes from the specific finding a probe recorded, not from this table.

* Email OTP is scored hard by default because a bare API/browser agent has no inbox access; an operator with an email-reading tool available would soften this in a per-deployment override, not in the base rubric. ** ToS bans are tagged per_transaction because the prohibition applies to every future attempt, not a single mechanical step — combined with the disqualifier rule below, this is what sinks a tool’s badge regardless of its numeric score.

Timing and hardness are properties of the finding, not the type: the same type (a CAPTCHA, a redirect-to-vendor) can appear at signup (setup, usually softer) or at every checkout (per_transaction, much worse). The probe records where in the flow it occurred — the multiplier tables below are what actually encode “recurring is much worse than one-time.”

How much each interrupt costs: the multiplier tables

Every finding’s penalty is base weight × timing × hardness × evidence quality. Three independent multiplier tables:

Timing

per_transaction (8×) vs. setup (1×) is the single biggest lever in the formula — it directly encodes “paid every use is much worse than paid once and amortized.” ongoing (3×) sits in between because it doesn’t hit every transaction but does recur indefinitely (fraud step-ups, rate-limit escalations).

Hardness

hard (2.5×) vs. soft (1×) encodes “cannot proceed at all” vs. “slower but still automatable.”

Evidence quality (probe tier)

A Tier-0 heuristic (regex over ToS text, a homepage challenge that might be unrelated to signup) is discounted relative to a live Tier-2 observation, which is ground truth. Documented evidence (strong public-record evidence of real end-to-end behavior, without our own probe run) sits between T1 and T2.

The scoring algorithm, in plain words

1. Start at 100.
2. Deduplicate. If more than one probe tier flags the same underlying interrupt (same type and same location in the flow — e.g. Tier 0 and Tier 1 both catching the same CAPTCHA, or a re-probe), only the highest-tier (most trustworthy) finding counts toward the score. Lower-tier duplicates are kept and shown as corroborating evidence, but excluded from the sum — they are never triple-penalized.
3. Sum weighted penalties. For each surviving finding, subtract base weight × timing × hardness × evidence quality from the running score.
4. Clamp to 0–100 — a tool can’t score below 0 or above 100.
5. Compute confidence from which probe tiers actually ran (see Confidence below).
6. Compute the badge — disqualifiers are checked first, independent of the numeric score; only if none apply do the score bands decide (see Badge tiers below).

Badge tiers

Why disqualifiers override the score

A tool could theoretically have a low score from minor accumulated friction but no single catastrophic blocker — that’s genuinely “moderate,” and the number should say so. But a tool with an explicit contractual ban on bots, or a CAPTCHA on literally every checkout, is categorically unusable for an autonomous agent regardless of how clean everything else is — no averaging should be able to dilute that into a passing grade. That’s why the three disqualifiers above (ToS bot-ban, contact-sales-only, any hard per_transaction finding) are checked first, before the score bands are even consulted, and force Blocked outright. A setup-time hard blocker (KYC, a manual approval queue) does not force Blocked, but it does structurally prevent Zero-Interrupt — it caps a tool at Low-Interrupt no matter how high the number is.

Confidence: how sure are we?

Every score ships with a confidence label reflecting how the evidence behind it was gathered:

Staleness decay: flows change — ChatGPT Instant Checkout itself flipped models within about six months. Confidence decays 0.15 for every 90 days since the last probe, floored at 0.20, so a score that hasn’t been rechecked in a while is shown as progressively less certain even if the badge hasn’t changed.

Probe tiers — how the evidence is gathered

Companion signals (displayed, never scored)

Two things affect how usable or trustworthy a tool is without being a “forced human intervention” in the sense the Interrupt Score measures. Both are shown on every tool page, but neither ever adjusts the score or badge above.

Machine-Readability Score

A separate 0–100 number covering how easy the integration itself is: presence of an OpenAPI spec, an llms.txt/agent-manifest, documented rate-limit headers, and structured error-response quality. Its absence doesn’t force a human into the loop, so it doesn’t belong in a metric literally defined as forced human intervention — but it’s still useful, so we report it alongside every score. Where a probe recorded a per-check breakdown, each individual check (pass/fail/not-checked) is shown on the tool page; hand-seeded demo listings show the score only.

Every machine-readability check carries an evidence grade — our own honesty label for how strong the real-world evidence is that the signal changes agent behavior. Proven: measured consumption or controlled-experiment effect (e.g. retrieval fetchers honoring robots.txt; AI fetchers not executing JavaScript). Plausible: a strong mechanism or documented failure mode, but no direct measurement (e.g. accessibility-tree semantics for browser agents, MCP servers). Emerging: publisher adoption is growing but no production agent consumes it yet — llms.txt is the canonical case: adoption grew ~8.8x in twelve months, yet a May 2026 log study across 137,000 domains found 97% of llms.txt files received zero requests. We keep scoring emerging standards (they are cheap to publish and adoption trajectories can flip fast — we only retire a check when its standard is dead, deprecated by its own steward, as ai-plugin.json was in 2024), but we label them honestly, and the grade never weights the score — the same posture as the T0/T1/T2 confidence tiers on the interrupt side. This model (machine-readability v3) also retired the ai-plugin.json check: OpenAI plugins were deprecated in 2024, and a review site scoring a dead standard would be exactly the failure mode these grades exist to prevent.

Custody / Risk Flag

custodial / non_custodial / not_applicable / unknown — whether a vendor holds funds/assets on the operator’s behalf. This is a risk dimension, not a friction dimension: a tool can have zero interrupts and a custodial ledger with no spend cap, or zero interrupts and non-custodial guarantees — conflating the two would corrupt the Interrupt Score itself, so it’s reported separately and rendered immediately adjacent to the badge on every tool page — never just linked from a methodology page — so a Zero-Interrupt badge is never mistaken for a safety endorsement it was never meant to make.

Two things we want to be upfront about

For how we make money and what a vendor can and can’t influence, see How we make money.