Reading the evidence

What each task state means, where the evidence lives in git, how the diff audit, acceptance strength, mutation testing and the gaming monitor work, and how to inspect any attempt.

The report

speccoder report               # every task
speccoder task show T004       # one task and its evidence

report reads only git, so it works on any clone. It also traces requirements (FR-001, SC-002) to the tasks that cite them (Trace 9/11 requirement(s) cited by a task; not cited: FR-007, SC-003), counts ticks without an attestation, and marks stale ticks.

When the spec changes

A tick is proven against the spec as it was at the task’s commit. When a requirement it cites is reworded, added or removed afterwards, the tick is stale (spec changed: FR-001 reworded since the tick).

• speccoder compares each tick’s commit with your committed HEAD: the feature’s documents (the task list aside) and shared ones such as the constitution. Uncommitted edits and other features do not count.
• A requirement is cited when its ID is a word in the task’s title.
• Stale (a cited requirement changed): the verdict stays; the report, the board, the inspector and the pull request mark it, and the requirements matrix stops counting it as proven. Changed elsewhere (the plan, the constitution, an uncited part of the spec): shown, not counted.
• speccoder run --converge re-checks: convergence is told which requirements changed and adds tasks for what no longer holds. speccoder never unticks a task on its own.
• In place with an ignored specs/, the commits’ Intent-Digest tells that the intent changed, not where.

What counts as red

Acceptance tests are evidence only if they failed before the task’s work existed. speccoder reads the runner’s report (JUnit, or the summary of libtest, nextest, pytest, jest, vitest and go test):

Anything else is not demonstrated (another build or syntax error, a collection error, a timeout, no tests collected, unrecognised output): the task can then be at most regression_only, and the report quotes the first lines of the error.

• Red is confirmed by a second run; red once and not again is flaky, not red.
• A check that fails, then passes on a rerun, is flaky: it does not fail the task, and a flaky task-specific check is never evidence for verified.
• If writing the acceptance tests fails, nothing is attempted and the task stops for you.

Tests that never ran

A test that never ran is evidence of nothing. After every attempt, each test file it added or changed (a test path that defines tests: #[test] and #[<crate>::test], def test…, it(/test(, func Test…) is checked against what the fast checks collected — a test counts only if a check the verdict rests on runs it, and ignored or skipped tests never count. In this order:

1. The toolchain’s static rule, which alone can prove a file is never collected: Cargo builds tests/*.rs and tests/<dir>/main.rs and the modules they declare (mod, #[path], autotests, [[test]]); pytest collects python_files under testpaths, outside norecursedirs; vitest and jest collect their include/testMatch (read from package.json or a JSON config, never from a config written as code); Go runs *_test.go outside testdata/ and _/. directories, under the build tags the check sets, with functions named Test…, Example…, Fuzz….
2. The check’s JUnit report, when it was freshly written by this run.
3. A listing built from the command of a fast check that runs the toolchain — its packages, targets, paths and filters kept: cargo nextest list, cargo test … -- --list (with --ignored to leave ignored tests out), pytest --collect-only, vitest list, jest --listTests, go test -list — or the check’s own list command; sandboxed, under the check’s deadline. A listing that changes the worktree fails the attempt.

A file from which no test ran fails the verdict with the rule as the reason, and the retry is told the file, the rule and the ways out:

tests not collected: `tests/unit/greet_name.rs` — Cargo builds `tests/*.rs` and `tests/<dir>/main.rs` (and the modules they declare); this file is neither

A file none of the three can decide stops the task, never ticked (TESTS-COLLECTION-UNKNOWN): one no fast check runs, one with a skip a listing cannot see (pytest.mark.skip, it.skip, t.Skip), or one the static rule cannot settle. Declare a JUnit report or a list command for the check, or speccoder task accept the task (it puts the saved attempt back before ticking).

• Test helpers without tests, deleted files and build output are not judged.
• A JUnit report counts only when this run wrote it: an older one is removed first, and one tracked by git is never read.
• The evidence shows a Collection row; the signed statement carries the files and what decided each.

Acceptance and held-out tests count at green only when the green run collected at least as many tests as the red run — or, when red reported no count (it did not build), as many as the files declare — and at least one. A green run whose count cannot be read is no evidence.

Why a tick is regression-only

Every regression_only tick carries one reason, decided by code, recorded in the commit (Regression-Only: trailer) and its signed statement, and worded the same everywhere:

For already_passing, speccoder runs the acceptance tests on the feature’s earlier ticks (bisected, at most five sandboxed runs, each in a scratch worktree of its own, under the checks’ deadline) to find which task implemented the behaviour, or that it passed before the feature. For a section added to a story’s file, only that section’s tests run (Rust, pytest and Go; for others no answer is given). Any run that cannot say — a worktree that cannot be made, a timeout, output that cannot be read — ends the search with “could not be determined”, never a task’s name. A feature with at least three ticks of which more than half are regression-only gets one Little Was Verified item in Needs You, with the reasons by count and the fix, until its branch is pushed.

Held-out tests

Frozen visible tests stop an implementer from editing the tests, not from tailoring code to them. So after the visible tests are red, a second author call writes held-out tests for the same scenarios, with other inputs and the edge cases the spec states. They must be red too.

After an attempt passes the visible checks, speccoder writes the held-out tests over the final tree, runs them in the sandbox and restores the tree. verified needs both. A failing held-out test fails the verdict; the retry is told only the scenario (held-out acceptance failed: US1-2 ("Given …")), never a path, test name, input or output.

The last task of a story with two or more scenarios also gets a composition test that exercises them together. [acceptance] held_out = false turns held-out tests off; the commit says Held-Out: off.

Where they are kept

Held-out tests never enter the worktree (except overlaid while a judge run checks them), the project’s repository or any prompt the implementer gets; the app, the CLI and the events show only their count, state and digest. Since every attempt can read the project’s refs and objects, they live in speccoder’s state directory ($SPECCODER_STATE_DIR, else $XDG_STATE_HOME/speccoder, else ~/.local/state/speccoder), under the repository’s own directory:

Publication. refs/speccoder/held-out/<feature> carries exactly the sets whose manifest digest a task closed on the pushed head names in its signed evidence (or Held-Out: trailer); sets of open tasks stay local. A feature run builds it when every task is ticked; speccoder pr builds and pushes it from the private store, never forced, writing nothing into the project’s repository. Each publication is parented on the branch and the previous one, so the ref only grows.

External agents ([executor]) run with your permissions, so the set is withheld, not sealed: it lies outside the worktree, where speccoder refuses every read the agent asks for, and a held-out literal copied into the attempt stops the tick.

Where the evidence lives

Each passed task is one commit on speccoder/<feature>, with trailers:

speccoder: T002 Implement `compare(…)`

Spec-Task: 001-semver/T002
Verdict: verified
Red: assertion
Acceptance-Strength: 0.83 mutants=9/11 near-miss=1/1
Run: run_01M5…
Usage: input=40250 output=3383 cache_read=30976 cache_write=0
Mutation: caught=26 missed=13 timeout=0 unviable=6

When they apply:

Each tick commit also carries signed statements of the same evidence in a git note (attestations).

After a run ticks the feature’s last task, the full checks run once; the result is recorded under refs/speccoder/full/<feature>, and speccoder pr refuses a branch whose full checks failed unless --force.

Every attempt that did not pass is a commit under refs/speccoder/attempts/… (git for-each-ref refs/speccoder/attempts/), then undone in the worktree; its message holds the outcome, reason, run and usage, and git diff speccoder/<feature> <ref> shows what it changed. Replay re-derives any recorded run without a model.

Tests decided before the run. The run report’s acceptance section names where the visible tests came from: authored (written by the run), review (frozen at review), bug_confirmation or person_test (bug fixes). Pre-decided tests carry their confirmation’s digests in pre_frozen and on the Acceptance: trailer; for review, the approval digest is the one review.json records, so a reviewer can match the tick to the committed sign-off.

The diff audit

Whatever the checks say, a verdict fails when the attempt:

• deletes or rewrites a test file that existed before it (adding tests can be allowed: see frozen tests);
• removes more test functions than it adds (the finding names them);
• adds skip markers (#[ignore], @pytest.mark.skip, .skip(, xit(, …);
• changes anything that decides verification: speccoder.toml, CI configuration, test-runner configuration;
• changes agent configuration (.claude/, .agents/, .codex/, .gemini/, .cursor/, .kiro/, .mcp.json, …), whose hooks and settings run code in a later agent session;
• changes an agent context file (AGENTS.md, CLAUDE.md, GEMINI.md) without a person’s approval.

The audit reads the diff, so it catches what a command, an MCP server or an external agent wrote, including files git ignores (an agent loads .claude/settings.local.json). These inputs are also re-checked after every command: a command that changes one ends the attempt at once.

Acceptance strength

Red → green shows the task’s work made the tests pass, not that they would fail a wrong implementation, and many model-written tests would not (SWE-Mutation, Beyond Fail-to-Pass). So after every verified verdict speccoder runs wrong implementations against the acceptance tests:

• near misses: variants of the changed code ([acceptance] near_miss, default 2, at most 3), written by a separate call that sees the spec and the code but never a test; each is wrong about exactly one scenario;
• mutants: with [mutation] configured, the tool’s mutants of the changed code, judged by the acceptance command alone.

A variant the tests fail on an assertion or a timeout is rejected; one that does not build is unviable and not counted; one that passes survived. Strength is rejected / judged:

Strength   0.77 mutants=9/11 near-miss=1/2
  survived: near-miss-2 (US1-2: an empty name is accepted)

• It is recorded on the tick (Acceptance-Strength:), in the report, the attestation and --json (strength); with held-out tests, as two numbers: visible alone, and visible plus held-out (what the verdict rests on).
• Survivors’ patches stay in the state directory; none reaches the implementing model.
• [acceptance] strength = false turns it off.

Strength changes a verdict only with [acceptance] min_strength (for example 0.8). A set below it is strengthened once: the test author is told what got through and adds tests, which must fail without the change and pass with it; then it is measured again. Still below, or not measurable (no variant, a tool error, the budget ran out), the task stops: stopped[ACCEPTANCE-WEAK].

Over-strict tests

The variant author also writes alternatives the spec permits ([acceptance] permitted, default 1; 0 turns this off): another order where the spec fixes none, other wording, another tie-break. If the acceptance tests reject one on an assertion, they may encode a choice the spec did not make. The task stops with stopped[TESTS-OVER-STRICT] (the scenario, the alternative’s patch, the failing test), and its attempt is held: its tree stays, the feature’s other tasks wait.

speccoder task rule T004 --tests-right --reason "the plan fixes the order"   # the tests are right
speccoder run --task T004                     # judges the held attempt again, no model call, and ticks it
speccoder run --task T004 --fresh-acceptance  # the tests are too strict: write them anew, attempt again
speccoder task accept T004 --reason "…"       # tick it on your word (attested)

task rule --variant permitted-1 rules on one finding; without it, on every finding held. A ruling is bound to that alternative’s hash and recorded, with who made it, on the tick (Ruling:) and in the attestation. The implementing model is never told.

Mutation testing

With [mutation] tool = "cargo-mutants", speccoder mutates only the code the passing attempt changed and runs the tests against each mutant; a mutant the tests miss is behaviour nobody checks. Strength asks whether the acceptance tests tell right from wrong; mutation testing asks whether all your tests check the changed code.

• The report shows caught and surviving mutants; the commit carries Mutation:.
• mode = "report" (default) records it; mode = "require" fails the verdict on survivors and tells the retry which survived (strict: some mutants are equivalent to the original).
• Other languages: tool = "command" with any tool that writes the standard mutation-testing JSON report (Stryker, Infection, PIT); see configuration.

The gaming monitor

With [monitor] enabled = true, a separate model call reviews every passing attempt. It sees the task, the diff and the tool calls (never the implementer’s explanation) and looks for hard-coded results, special-cased inputs, weakened tests and similar shortcuts. A suspicion keeps the task unticked (suspected_gaming) until you decide. It only adds doubt: it never makes a failing task pass, and the implementer never learns what it found.

What the evidence does not show

• verified means no more than the checks and tests you configured and the examples in your spec.
• A strength of 1.0 means the tests rejected every wrong variant tried, not every possible one; a surviving near miss may be one its author got wrong.
• A spec without acceptance examples yields regression_only; a wrong example yields a wrong test.
• What speccoder cannot demonstrate it reports as such (not_demonstrated, blocked), never guessed.

Edit this page