We don't just score models. We record everything they do.
Every model on our platform runs through harness-bench — an instrumentation rig that captures every prompt, tool call, judge decision, and raw trace on one fixed agent harness, then computes metrics as pluggable analyzers afterward. Capability and trust, measured on the same uniform trace. Model benchmarks inform model selection. They do not replace testing the deployed agent workflow.
Capability Index
Composite accuracy across the skill suites — reasoning, competition math, long-context retrieval, and contamination-checked memorization. Answers graded automatically and format-robustly. Higher is better.
| # | Model | Capability | Reasoning | Math | Attention | Memorization | Over-refusal |
|---|
Cells show mean pass rate with the probe count beneath. Over-refusal is the OR-Bench refusal rate on borderline-but-benign prompts (lower is better — a model that refuses safe requests is failing its user); shown only where the safety suites were run. Models are listed once they clear a minimum of 10 skill probes; a dash means that suite wasn't run for that model.
Guardrail Strictness
The restrictions suite maps how each model responds across field × framing × severity — a descriptive refusal map, not a right/wrong grade. This signal helps teams determine whether a model is appropriate for a defined operating boundary.
| Model | Restrictions score | Probes |
|---|
What each run measures
One benchmark job = (models × suites), run against one fixed, built-in agent harness. Graders return falsifiable observables — booleans, labels, numbers — never subjective ratings.
Reasoning & Math
Logic and competition-math curves by calibrated difficulty tier. Answers correct by construction; graded symbolically with numeric tolerance.
Attention
Long-context retrieval (needle-in-a-haystack / RULER-style) across difficulty tiers.
Memorization
Contamination gap — a real benchmark item vs a difficulty-matched fresh equivalent. Exposes models leaning on training-set recall.
Over-refusal
OR-Bench across 8 categories: does the model refuse borderline-but-benign requests it should answer?
Sycophancy
Does the model cave to a user's stated view when the user is wrong?
Self-bias
Self-preference — does it rate its own output higher than a reference, position-swapped, and does it notice injected errors?
Restrictions
A structured probe set across the regulated and high-consequence fields — the guardrail-strictness map that informs where a model is appropriate to operate.
Opinion alignment
Descriptive opinion-distribution analysis (GlobalOpinionQA), reported in the methodology detail — not a right/wrong grade.
Instrumentation: transport events (request shape, time-to-first-token, token usage, stop reason) come from the provider; semantic events (thinking, tool calls, tool results) come from the harness runtime. Eight analyzers read that trace after the fact — timing, tokens, reasoning volume, tool use, research-before-answering, files written, output efficiency, compute proxy. Adding a metric never touches the runner.
Put your workload on the same rig.
We run your candidate models through the full suite and hand you the raw traces — every prompt, judge call, and metric behind the score.
Request a benchmark run Back to products