Benchmarking the capability spectrum and reliability of System One models

Verified labels
Difficulty tiers
Agent sandboxes
Reliability tests

Lichen Li^*, Ziping Dong^*, Sicheng Feng, Qi Li, Xinchao Wang^†

* Equal contribution · † Corresponding author

View the leaderboard Paper Code Dataset

01 — At a glance

Capability, speed and reliability

02 — Leaderboard

Leaderboard

The mean of 11 domains, each on its hardest tier.

All results (JSON)

03 — Decision reliability

Does a decision hold when the options change?

SOption-order stability: keeps its answer when the options are shuffled NNull discrimination: notices the missing answer RReliability = √(S·N), from 0 to 100

Reliability ranking

04 — Domains

Where System One matches System Two, and what closes the gap

How big is the gap to System Two, and where is it largest?

05 — The benchmark

2,919 tasks in 11 domains

Static questions plus web and tool sandboxes, all posed as decisions.

How the leaderboard is scored

Domains count equally, each on its hardest tier: 1,232 tasks.

C (m) = \frac{100}{D} \sum_{d = 1}^{D} \frac{1}{| T_{d} |} \sum_{i \in T_{d}} c_{i} (m)

c_i(m) ∈ [0, 1] is model m's credit on task i; T_d holds domain d's leaderboard tasks; D = 11.

06 — Evaluate

Evaluate your model

Three steps, any model.

Step 01 · Get the data

git clone https://github.com/holdrain/SO-Arena
cd SO-Arena
./jev fetch-data --output /path/to/so-arena-data
export SO_ARENA_DATA=/path/to/so-arena-data
./jev verify
./jev setup

Downloads and checks the data.

Step 02 · Run your model

export JEV_ENDPOINT=http://localhost:8000
./jev run --adapter adapters:http \
  --model your-model --out /path/to/new-run

Point it at your model server.

Step 03 · Score

./jev score --run /path/to/new-run \
  --output /path/to/report.json

The leaderboard score, by domain and tier.

Full instructions: the SO-Arena repository.

07 — Citation

Cite SO-Arena

Correspondence: xinchao@nus.edu.sg

@misc{li2026soarena,
  title         = {SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models},
  author        = {Lichen Li and Ziping Dong and Sicheng Feng and Qi Li and Xinchao Wang},
  year          = {2026},
  eprint        = {2611.15777},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2611.15777}
}