Tbench AI
Methodology v1.0 · verifier assurance

Your benchmark is only as trustworthy as its verifier.

Tbench validates the task, environment, reference solution and verifier — not the model. One task or a complete set enters the same evidence-bound workflow, and a required stage that did not run never becomes a pass.

Input
One task or a selected set
Decision basis
Required coverage before score
end-to-end workflow stages
14
registered static rules
47
explicit verdict states
5
engine tests collected · separate evidence
704

Engine inventory snapshot: 2026-09-21. This is release-level collection evidence, not the number of tests executed for an individual submission.

Loading the published methodology catalogue…

What the number means

Fourteen stages around one evidence boundary.

The workflow is one source lock, twelve task-level validation stages and one submission-level evidence seal. The regression tests qualify the Tbench engine; they are not hundreds of checks claimed against your task.

All fourteen stages apply to a single-task audit: one source lock, twelve task-level stages, and one evidence seal. There is no minimum batch size.

SourceContractExecutionAdversarialAttestation
  1. 01Source lock and provenanceSource integrityRequireddeterministicOnce per audit
    Binds the repository revision, task tree digest, tool version and audit policy.
  2. 02Contract and scaffold auditStatic architectureRequireddeterministicOne task
    Runs the registered rules against structure, metadata, instructions, launchers and verifier contracts.
  3. 03Requirement traceabilityRequirement mappingRequireddeterministicOne task
    Maps graded requirements from the instruction through the solution to verifier assertions.
  4. 04Novelty and contamination screenCorpus integrityRequireddeterministicOne task
    Compares the task with a pinned corpus rather than treating a single task as vacuously unique.
  5. 05Oracle and no-op calibrationContainer calibrationRequiredcontainerOne task
    Runs positive and negative controls in isolated images.
  6. 06Pass@2 precheckDifficulty calibrationRequiredmodelOne task
    Runs the fixed two-attempt gate and separates solver failure from infrastructure failure.
  7. 07Verifier boundary auditVerifier rigorRequiredcontainerOne task
    Challenges false accepts and false rejects across invalid and boundary-case solutions.
  8. 08Fresh-input witness verificationIndependent re-executionRequiredcontainerOne task
    Re-executes the deliverable on fresh seeded inputs.
  9. 09Mutation sensitivityMutation rigorRequiredcontainerOne task
    Introduces controlled defects and measures whether the verifier kills them.
  10. 10Multi-model difficulty probeDifficulty evidenceRequiredmodelOne task
    Preserves panel disagreement and empirical difficulty evidence.
  11. 11Reward-hacking challengeAdversarial resistanceRequiredcontainerOne task
    Attempts leakage, protected-path access and shortcuts that earn reward without doing the task.
  12. 12Pass@5 frontier gauntletEmpirical gauntletRequiredmodelOne task
    Records each valid attempt, model, environment and observed outcome.
  13. 13Grounded coherence reviewAutomated reviewAdvisorymodelOne task
    Adds an advisory second opinion without overriding deterministic evidence.
  14. 14Evidence seal and submission decisionAttestation and registerRequiredaggregateOnce per audit
    Binds stage records to exact digests and issues or withholds the submission result.
Rule catalogue

The checks inside the stages.

Forty-seven registered static rules are grouped by failure mode. Forty-five are active under the standard policy; two witness checks support fresh-input re-execution.

A

Reference solution correctness

Validates that the reference solution computes the graded artifact legitimately.

3 rules
B

Contract coherence

Finds missing, ambiguous, or undisclosed graded requirements.

3 rules
C

Verifier rigor and scoring

Tests graded coverage, tolerances, scoring semantics, and judge gating.

9 rules
D

Determinism and fixtures

Checks fixtures, randomness, clocks, networks, and reproducible execution.

4 rules
E

Isolation and anti-cheat

Detects answer leakage, writable trust boundaries, and unsafe file handling.

7 rules
S

Scaffold and platform contract

Validates task layout, launch behavior, metadata, reports, and container contracts.

19 rules
W

Witness verification

Checks independent re-execution on fresh witness inputs.

2 rules
Verdict vocabulary

Five states. No convenient sixth.

The most important distinction is between a task defect and an audit that could not complete. Both can withhold a result, but they demand different remedies and stay different in the evidence.

  1. 01Passed

    The stage ran, produced evidence, and satisfied its published rule.

  2. 02Failed

    The stage ran and established a task or verifier defect. A required failure withholds the mark.

  3. 03Needs review

    Evidence exists, but a person must decide. It is neither silently passed nor collapsed into an error.

  4. 04Did not run

    A prerequisite or tool was unavailable. Nothing was established, so a required skip blocks issuance.

  5. 05Errored

    Infrastructure failed before a verdict existed. The report names it separately from a defect in the task.

Decision rule

Coverage comes before the score.

A submission with partial required coverage is not partially certified. Stage results remain visible, but the public result is withheld until the required workflow is complete.

  1. 01

    Tbench Certified

    Every required stage ran and passed. The evidence seal binds the result to the submitted source and methodology revision.

  2. 02

    Conditional

    Every required stage ran, but evidence needs a named human decision. The limitations travel with the result.

  3. 03

    Not issued

    A required stage failed, skipped or errored. The record remains useful because it identifies the blocker and remediation path.

Bring the verifier, not a pitch deck

Audit one task. Scale when the evidence holds.

Start with a single PR or submit a full set. The unit changes; the standard does not.