Proof in pull requests

speccoder attest verify --branch checks every ticked task of a branch against its signed evidence, with the maintainer's trust: no model key, no network, nothing from the pull request executed. The speccoder / evidence check, the evidence tour and the badge.

speccoder attest verify --branch checks a branch’s attestations for a maintainer: no model key, no network, no trust in the contributor’s machine. On GitHub, the evidence action (packaging/speccoder-verify-action) downloads the speccoder release, runs it on every push to a pull request and publishes the check speccoder / evidence.

What it checks

The check takes every unit (feature, change) whose task list or intent files differ between the head and the merge base with --base, in full (so a spec edit that stales an older tick is caught), and for every [X] there:

1. finds the latest commit reachable from the head that carries the task’s verdict (Spec-Task: <unit>/<task> and Verdict:);
2. checks that commit’s attestation with the rules of speccoder attest verify (one implementation, so both agree);
3. checks what only a branch shows: a tick no commit carries evidence for, a task line edited after its tick, a cited requirement changed since (drift), evidence on a commit a squash merge dropped.

Reasons for unverifiable: ticked_without_evidence (ticked by hand), missing (no note: push refs/notes/speccoder), malformed, bad_signature, mismatch (also a task line reworded after the tick), untrusted_signer, unsigned, commit_unreachable (a squash or rebase merge lost the commit), tool_missing (ssh-keygen, gpg or cosign cannot run here).

The branch’s conclusion is failure when any tick fails, success when every tick carries verifying evidence, and neutral when there is nothing to check (no ticks in the touched units, or no spec project in a tree read whole); neutral is never success.

A check that cannot read what it checks is refused (a non-zero exit, no check payload), never neutral, so --allow-neutral cannot pass it: a failing git command, a tree that cannot be written out whole (an unreadable blob, two paths equal but for letter case), an unreadable unit, or a head that drops the spec project its base (or the trust’s revision) holds. Skipped tasks are listed, not counted. Regressions of earlier tasks by later ones are not evaluated, and the check says so.

Run it

It is part of speccoder (installation).

speccoder attest verify --branch --base origin/main \
  --allowed-signers-rev origin/main:.github/speccoder/allowed_signers

failure: 7 verified, 1 regression only, 1 attested, 1 stale, 2 unverifiable
  007-config/T012  verified
  007-config/T013  unverifiable: ticked_without_evidence — no commit carries this task's verdict (ticked by hand?)
  007-config/T014  stale — FR-004 changed since the tick: the verdict is for the earlier wording

Trust comes from you, never from the branch. Without one of these (or git’s gpg.ssh.allowedSignersFile on your machine), it refuses:

• OpenPGP signatures are checked against the caller’s keyring (GNUPGHOME): one good signature, by the key gpg validated, that the keyring trusts fully or ultimately, read from gpg’s status lines by keyword (a user ID that says TRUST_ULTIMATE trusts nothing). Every such key is trusted, so keep the keyring to the project’s signers and leave auto-key-retrieve off. The signer shown is the key’s fingerprint.
• Unsigned evidence is unverifiable unless you pass --require-signed=false; the report then says unsigned ticks were accepted.

It executes nothing from the checked tree. It reads git objects only, never a working tree, and spawns only git and the signature tool a signature names (a test runs it with nothing else on the PATH), with an environment holding nothing but PATH, HOME, GNUPGHOME, LANG, TMPDIR and GITHUB_STEP_SUMMARY: speccoder runs itself again with only those before it starts. It calls no model and no HTTP client.

Exit status of attest verify --branch: 0 success, 26 failure, 15 neutral, 2 a refusal (no trust, a bad flag), 1 an error. attest tour and attest badge exit 0 when they produced their output.

The check on the pull request

speccoder / evidence is created on the pull request’s head commit on every push. Its title gives the outcome and counts; its summary the verdict bar (verified 7 · regression only 1 · attested 1 · stale 1 · unverifiable 2), the trust used and what was not evaluated; its text every tick, the requirement trace and each task’s tour. Annotations:

GitHub takes 50 annotations per request (the rest follow in updates) and 65 535 characters of summary and of text; what does not fit is counted, never silently cut, and the conclusion covers every tick. With --github-annotations, at most 10 of each level are printed, and a notice counts the rest.

The evidence tour

Each tick reads as six steps, each labelled with its source: signed (a signed statement says it), committed (git objects) or not recorded:

1. Requirement: the passage the task cites, as the tick’s commit holds it;
2. Frozen tests: the acceptance tests, and where they were decided;
3. Red: which tests failed first, how (assertion, missing symbol), and the signed excerpt of the output;
4. Diff: the task commit’s changes;
5. Green: what passed after: acceptance, held-out, the checks against their baseline;
6. Strength: how many wrong variants the tests reject.

A step without evidence says so in the verdict’s words (REGRESSION ONLY — nothing task-specific was shown, ATTESTED — a person's word, not evidence). An unverifiable tick shows only its requirement and the reason. The tour is in the check, in speccoder pr’s description, in the app’s task inspector (Tour), and on its own:

speccoder attest tour T012 --feature 007-config           # Markdown
speccoder attest tour T012 --feature 007-config --json    # payload evidence_tour

The badge

speccoder attest badge --allowed-signers-rev main:.github/speccoder/allowed_signers \
  --svg speccoder-evidence.svg --endpoint speccoder-evidence.json

The share of ticks, over every unit, whose evidence verifies and is not stale (skipped tasks excluded; none without ticks): green from 95 % with nothing unverifiable, yellow from 75 %, else red. --endpoint writes a shields.io endpoint file. The action publishes it only when asked, only from the default branch (badge-branch). What a tick signs: the task-verification predicate.

Edit this page