We don't just score models. We record everything they do.
Every model on our platform runs through a harness-bench: an instrumentation rig that captures every prompt, tool call, judge decision, and raw trace on one fixed agent harness, then computes metrics as pluggable analyzers afterward.
Capability Index
Composite accuracy across the skill suites — reasoning, competition math, long-context retrieval, and contamination-checked memorization. Answers are graded automatically and are robust to formatting differences. Higher scores indicate better performance.
| # | Model | Capability | Reasoning | Math | Attention | Memorization | Over-refusal |
|---|
Cells show mean pass rate with the probe count beneath. Over-refusal is the OR-Bench refusal rate on borderline-but-benign prompts (lower is better, since a model that refuses safe requests is failing its user); shown only where the safety suites were run. Models are listed once they clear a minimum of 10 skill probes; a dash means that suite wasn't run for that model.
Guardrail Strictness
The restrictions suite maps how each model responds across field × framing × severity — a descriptive refusal map, not a right/wrong grade. This signal helps teams determine whether a model is appropriate for a defined operating boundary.
| Model | Restrictions score | Probes |
|---|
What each run measures
One benchmark job = (models × suites), run against one fixed, built-in agent harness. Graders return falsifiable observables — booleans, labels, numbers — and never subjective ratings.
Reasoning & Math
Logic and competition-math curves by calibrated difficulty tier. Answers correct by construction; graded symbolically with numeric tolerance.
Attention
Long-context retrieval (needle-in-a-haystack / RULER-style) across difficulty tiers.
Memorization
Contamination gap — a real benchmark item vs a difficulty-matched fresh equivalent. Exposes models leaning on training-set recall.
Over-refusal
OR-Bench across 8 categories: does the model refuse borderline-but-benign requests it should answer?
Sycophancy
Does the model cave to a user's stated view when the user is wrong?
Self-bias
Self-preference — does it rate its own output higher than a reference, position-swapped, and does it notice injected errors?
Restrictions
A structured probe set across the regulated and high-consequence fields — the guardrail-strictness map that informs where a model is appropriate to operate.
Opinion alignment
Descriptive opinion-distribution analysis (GlobalOpinionQA), reported in the methodology detail rather than as a right/wrong grade.
Instrumentation: transport events (request shape, time-to-first-token, token usage, stop reason) come from the provider; semantic events (thinking, tool calls, tool results) come from the harness runtime. Analyzers read that trace after the fact, capturing timing, tokens, reasoning volume, tool use, research-before-answering, files written, output efficiency, compute proxy.