Method

The verifier recomputes the truth from the task. There is no stored answer key to leak.

Exact grading

Verifiers use exact Fraction arithmetic. Floats and booleans in answers are rejected, so there is no tolerance to game. Where a family is checked by a certificate rather than a re-solve, the certificate is verified exactly.

One strict JSON answer

The agent returns a single JSON object. Extra text, markdown fences, two objects, duplicate keys and NaN are rejected. Result codes are coarse, so the verifier output cannot be used to read off the answer.

Tamper detection

Each task has an instance hash. The verifier checks it and fails closed if the task was edited to match a wrong answer. The harness must keep the hash where the agent cannot write.

Secret seeds

Instances are generated deterministically from a seed. Using a private seed at evaluation time means the instances are not in any public set. The seed must be kept secret by whoever runs the evaluation.

Graded scores

The search and induction families define an exact score between 0 and 1 beside binary pass/fail; a pass means the score is exactly 1. The score formula is stated in the task text.

Sandboxed harness and containers

A sandboxed episode harness runs the agent without the verifier in view. Every episode type can also run inside a Docker container (--backend docker), and the harness checks after each episode that no container or stray process is left behind. Container runs were checked on one machine (macOS with Colima) using scripted agents against a fake model API. They have not been run with a real model or on native Linux, and bugfix and feature episodes were not among those checked.

Signed episode files

For released measurements, every episode in a results file carries a signature and the file has a signed header listing every task and rollout. When the file is graded, an unsigned, edited, duplicated, spliced or partly removed file is refused, so editing a results file cannot raise its pass count. This protects against everyone except whoever holds the release secret and runs the evaluation; it is integrity, not proof against the host.

Exports for other evaluation frameworks

Tasks can be exported for SkillsBench/BenchFlow and for Prime Intellect's verifiers. The exports were run through those projects' own tools. No model and no API key were used: the agent was either the package's reference solver or a deliberately wrong one.

• SkillsBench / BenchFlow: 8 exported tasks passed bench tasks check and the repository's lint and validation scripts. Running them with the reference solver in Docker gave 8 of 8 at reward 1.0. Four negative controls (a wrong answer, a no-op, an edited input) scored 0 of 4 with the expected failure codes.
• Prime / verifiers: vf-eval on two environments (Betti numbers and invoice reconciliation), 4 examples each, scored 1.00 with reference answers and 0.00 with wrong ones. The exported packages are self-contained and install without this repository.

Not done: interactive families have no exporter, no real model has been run through either export, and neither project has reviewed or accepted these tasks. Running their tools is not an endorsement by their maintainers.

Red-team

A scripted red-team log records what was tried and what was found. It covers scripted attackers on every family, and one model-driven black-box attempt (12 grading calls per task) on the four topology and graph families, which found no exploit. On 2026-10-27 a scripted red-team of the interactive families (flaky sync, clarify-and-report and the interactive release path) was run by an AI model with scripts. It confirmed 6 issues, including a forged results file, a bias in how clarify instances were screened and a free probe in flaky sync, and all 6 were fixed the same day. An AI model told to cheat has not been run. The finance, science, coding and back-office families have not had a model-driven red-team yet. This is not a proof of cheat resistance.

Known limits

• Tasks are solvable by any agent with code execution, so the tool policy has to be stated for each evaluation.
• Container runs have not used a real model, and the namespace sandbox (used without Docker) is not a full container.
• Only a few families and levels have been run against any model, with few attempts each. See Difficulty.
• No blind human check of task fairness and no expert audit have been done.
• Instruction text for the finance and science families has not been read by anyone other than its author, so ambiguity there is untested.