Your benchmark is only as trustworthy as its verifier.
Tbench validates the task, environment, reference solution and verifier — not the model. One task or a complete set enters the same evidence-bound workflow, and a required stage that did not run never becomes a pass.
- end-to-end workflow stages
- 14
- registered static rules
- 47
- explicit verdict states
- 5
- engine tests collected · separate evidence
- 704
Engine inventory snapshot: 2026-09-21. This is release-level collection evidence, not the number of tests executed for an individual submission.
Loading the published methodology catalogue…
Fourteen stages around one evidence boundary.
The workflow is one source lock, twelve task-level validation stages and one submission-level evidence seal. The regression tests qualify the Tbench engine; they are not hundreds of checks claimed against your task.
All fourteen stages apply to a single-task audit: one source lock, twelve task-level stages, and one evidence seal. There is no minimum batch size.
01Source lock and provenanceSource integrityRequireddeterministicOnce per audit
Binds the repository revision, task tree digest, tool version and audit policy.02Contract and scaffold auditStatic architectureRequireddeterministicOne task
Runs the registered rules against structure, metadata, instructions, launchers and verifier contracts.03Requirement traceabilityRequirement mappingRequireddeterministicOne task
Maps graded requirements from the instruction through the solution to verifier assertions.04Novelty and contamination screenCorpus integrityRequireddeterministicOne task
Compares the task with a pinned corpus rather than treating a single task as vacuously unique.05Oracle and no-op calibrationContainer calibrationRequiredcontainerOne task
Runs positive and negative controls in isolated images.06Pass@2 precheckDifficulty calibrationRequiredmodelOne task
Runs the fixed two-attempt gate and separates solver failure from infrastructure failure.07Verifier boundary auditVerifier rigorRequiredcontainerOne task
Challenges false accepts and false rejects across invalid and boundary-case solutions.08Fresh-input witness verificationIndependent re-executionRequiredcontainerOne task
Re-executes the deliverable on fresh seeded inputs.09Mutation sensitivityMutation rigorRequiredcontainerOne task
Introduces controlled defects and measures whether the verifier kills them.10Multi-model difficulty probeDifficulty evidenceRequiredmodelOne task
Preserves panel disagreement and empirical difficulty evidence.11Reward-hacking challengeAdversarial resistanceRequiredcontainerOne task
Attempts leakage, protected-path access and shortcuts that earn reward without doing the task.12Pass@5 frontier gauntletEmpirical gauntletRequiredmodelOne task
Records each valid attempt, model, environment and observed outcome.13Grounded coherence reviewAutomated reviewAdvisorymodelOne task
Adds an advisory second opinion without overriding deterministic evidence.14Evidence seal and submission decisionAttestation and registerRequiredaggregateOnce per audit
Binds stage records to exact digests and issues or withholds the submission result.
The checks inside the stages.
Forty-seven registered static rules are grouped by failure mode. Forty-five are active under the standard policy; two witness checks support fresh-input re-execution.
Reference solution correctness
Validates that the reference solution computes the graded artifact legitimately.
Contract coherence
Finds missing, ambiguous, or undisclosed graded requirements.
Verifier rigor and scoring
Tests graded coverage, tolerances, scoring semantics, and judge gating.
Determinism and fixtures
Checks fixtures, randomness, clocks, networks, and reproducible execution.
Isolation and anti-cheat
Detects answer leakage, writable trust boundaries, and unsafe file handling.
Scaffold and platform contract
Validates task layout, launch behavior, metadata, reports, and container contracts.
Witness verification
Checks independent re-execution on fresh witness inputs.
Five states. No convenient sixth.
The most important distinction is between a task defect and an audit that could not complete. Both can withhold a result, but they demand different remedies and stay different in the evidence.
- 01Passed
The stage ran, produced evidence, and satisfied its published rule.
- 02Failed
The stage ran and established a task or verifier defect. A required failure withholds the mark.
- 03Needs review
Evidence exists, but a person must decide. It is neither silently passed nor collapsed into an error.
- 04Did not run
A prerequisite or tool was unavailable. Nothing was established, so a required skip blocks issuance.
- 05Errored
Infrastructure failed before a verdict existed. The report names it separately from a defect in the task.
Coverage comes before the score.
A submission with partial required coverage is not partially certified. Stage results remain visible, but the public result is withheld until the required workflow is complete.
- 01
Tbench Certified
Every required stage ran and passed. The evidence seal binds the result to the submitted source and methodology revision.
- 02
Conditional
Every required stage ran, but evidence needs a named human decision. The limitations travel with the result.
- 03
Not issued
A required stage failed, skipped or errored. The record remains useful because it identifies the blocker and remediation path.
Audit one task. Scale when the evidence holds.
Start with a single PR or submit a full set. The unit changes; the standard does not.
