Rating method v1
The rules the code enforces. Thresholds are editorial choices, published as such, and not yet validated. Source: the rating-method report.
Principles
- A board answers one decision. It declares the job, region, workload, budget and hard requirements. Scores never compare across boards. There is no universal winner.
- Evidence comes before summaries. Rows are append-only and each carries its source.
- "Insufficient evidence" is a normal result. The engine refuses to name a winner it cannot support.
- Price is a scenario, computed for the board's workload. It is never a list price divided into a score.
Evidence row
evidence/*.jsonl, one JSON object per line, never edited after commit. A correction is a new row with supersedes.
| Field | Meaning |
|---|---|
id | Stable id |
product, version | What was measured, down to the version |
metric, unit, value | The number. Missing stays missing |
uncertainty | {low, high, kind}, where kind is ci95, tolerance or unknown |
sample_units, repeats | Null when the source does not say |
protocol | Named protocol and version |
conditions | Language, region, tier and so on |
tested_at, retrieved_at | Dates |
source_url | Where it was read |
source_id | Key into the rights register |
origin_group | Rows sharing an origin count once |
claim_type | MEASURED, STATED or OPINION |
conflict | Known interest: vendor, affiliate, sponsored, none, unknown |
A vendor's own benchmark is STATED with conflict: vendor, and has zero weight in quality.
Rights register
rights/sources.json lists every source with status: cleared, attribution, facts-only, private or forbidden. The public build fails on any row whose source is not cleared or attribution, or facts-only for a price fact with a link. A source whose terms bar this use is forbidden, and one with no licence for its data is private. Neither is read into a public build.
Score
For each metric with frozen anchors a (unacceptable) and b (sufficient):
u(x) = 100 * clip((x - a) / (b - a), 0, 1)Never use the current catalogue's minimum and maximum: adding a product must not move another's score.
Unequal sources, after collapsing each origin_group to one row:
u_hat(i, k) = sum_s(r_s * u(i, s, k)) / sum_s(r_s)
Q(i) = sum_k(w_k * u_hat(i, k)), sum_k(w_k) = 1| Source protocol | r_s |
|---|---|
| Auditable, raw observations public | 1 |
| Documented, no raw observations | 0.5 |
| Unsupported claim, sentiment, or vendor self-test | 0 |
Sources never gain weight for agreeing with us, audience size or review count.
Freshness: r_s(t) = r_s(0) * 2^(-t/h), with h = 30 days for AI services and 365 days for hardware. Evidence older than 2h is excluded. A new version needs new evidence. Age lowers the weight of the evidence, never the performance.
Uncertainty and abstention
- Raw repetitions: 95% bootstrap interval, clustered by listener, item and unit.
- No sampling information: the source's documented tolerance, else unknown.
- Several comparable origins: the envelope, labelled a sensitivity range, never a confidence interval.
- A missing metric counts as
[0, 100]. Its weight is never redistributed.
A product is named Best only if all of these hold:
- Its lower bound exceeds every competitor's upper bound by at least 3 points.
- Every critical metric is present, from two independent origins or one auditable own test.
- Its price was verified within the board's price limit: 30 days for tariffs, 72 hours for street prices.
Otherwise the board publishes an unresolved shortlist, the set that no other product beats with certainty, and says what evidence would settle it.
Views
| View | Rule |
|---|---|
| Map | Pareto frontier of cost against quality. A product is on the frontier if nothing is at least as cheap and at least as good, with one strictly better. Quality uses the point estimate; the bounds are drawn. |
| Ranking | Ordered by Q within budget, with the abstention result above it. |
| Value ratio | Only with a physical unit, for example successful tasks per euro. Never Elo per dollar. |
Cost
C(H) = delivered price + fixed fees + usage + maintenance - residual value, over horizon HFor AI tools: the cost of the board's workload at the cheapest plan that meets its requirements, including commercial rights, minimum commitment and overage. Published with low and base scenarios. Currency, seller and date are stored as read.
Reproducibility
Each publish writes snapshots/<date>/<board>.json with evidence ids, code commit, anchors, weights, prices and seed. Each board has a stable URL, a JSON twin and a changelog that explains every rank change.
Recommendations are not measurements
"Who recommends what" is a separate ledger: reviewer, product, job, reasons, date, URL, disclosed interest. LLM extraction is allowed only from allowlisted pages, treats page text as data, and is checked by a person before publishing. It never enters Q. A signal such as a vendor author or an affiliate link is stated as fact. Nobody is called paid or fraudulent on signals alone.
Anti-gaming
Identities hidden in listening tests. Held-out scripts rotated. Failed runs kept. The same public endpoint the customer buys is tested. Vendors may submit corrections with evidence and get no veto; corrections are logged.
Validation, before any claim of quality
Hidden repeat clips for test-retest reliability. Leave-one-origin-out and plus-or-minus 20% weight sensitivity on every board. Backtest against historical snapshots once they exist. Agreement with blinded experts in v2.
Left for v2
Learned source reliability, hierarchical pooling, Bradley-Terry over our own votes, representative satisfaction panels.