iw ci-gate reads a commit's whole status history, so a re-run can never go green #472

New issue

Closed

opened 2026-08-27 19:12:27 +00:00 by mph · 1 comment

mph commented

2026-08-27 19:12:27 +00:00

Owner

What happens

./iw ci-gate reports CI failed and writes status: ci_failed into the slice
or phase card for a pull request whose checks are all green, whenever the
commit has had more than one CI run.

Seen on PR #470 (commit 811feec). Run #151 failed three jobs because the
runner could not reach the maven mirror:

Error while downloading https://maven.iterative.works/maven-central/ch/epfl/scala/scalafix-cli_3.8.4/0.14.7/scalafix-cli_3.8.4-0.14.7.jar: Network is unreachable, ignoring it
⚙️ [runner]: exitcode '255': failure

Re-triggering produced run #152 on the same commit, with all six jobs
success (compile, format, lint, test, dashboard-build, contract). ci-gate
still answered:

Error: CI failed — 3 job(s):
  test  (failure)
  lint  (failure)
  dashboard-build  (failure)

Why

A commit's status list is append-only: each run adds a row per context rather
than replacing one. The forge itself resolves this correctly —

GET /api/v1/repos/iterative-works/iw-cli/commits/811feec.../status
  → "state": "success", six contexts, each green

— but the per-status endpoint keeps the whole history:

GET /api/v1/repos/iterative-works/iw-cli/commits/811feec.../statuses
  → 38 entries, newest first:
      success  CI / test             19:05:16   ← run #152
      success  CI / dashboard-build  18:57:50   ← run #152
      ...
      failure  CI / test             18:52:xx   ← run #151, still in the list

ci-gate appears to scan all entries and treat any failure as the verdict,
instead of taking the latest status per context (or asking the combined
/status endpoint, which already does exactly this).

Consequence

Any close or phase gate that has ever had to re-run CI — for a flaky runner, a
network blip, a cancelled run — is stuck reporting failure permanently, and
writes a false ci_failed into the card the dashboard shows. The only way out
is to push a new commit so the status history starts clean, or to correct the
card by hand after checking the combined status.

A second, smaller defect in the same run: the gate wrote
ci_gated_sha: e1cf757 — the commit before the merge it had just judged, not
811feec.

Suggested fix

Resolve each context to its most recent status before deciding, or read the
combined /commits/{sha}/status endpoint, whose state is already the answer.
Stamp ci_gated_sha with the commit actually gated.

Neighbours

IW-437 covers the gate calling a slow pipeline failed (the wait budget
expiring against a green run). This is the gate calling a re-run pipeline
failed, which is a different cause with the same visible outcome — a green pull
request marked ci_failed.

Found while closing slice dashboard-control-plane/08-the-cleanup-gets-done
(PR #470); recorded in that effort's LESSONS.md.

## What happens `./iw ci-gate` reports `CI failed` and writes `status: ci_failed` into the slice or phase card for a pull request whose checks are **all green**, whenever the commit has had more than one CI run. Seen on PR #470 (commit `811feec`). Run #151 failed three jobs because the runner could not reach the maven mirror: ``` Error while downloading https://maven.iterative.works/maven-central/ch/epfl/scala/scalafix-cli_3.8.4/0.14.7/scalafix-cli_3.8.4-0.14.7.jar: Network is unreachable, ignoring it ⚙️ [runner]: exitcode '255': failure ``` Re-triggering produced run #152 on the **same commit**, with all six jobs `success` (compile, format, lint, test, dashboard-build, contract). `ci-gate` still answered: ``` Error: CI failed — 3 job(s): test (failure) lint (failure) dashboard-build (failure) ``` ## Why A commit's status list is append-only: each run adds a row per context rather than replacing one. The forge itself resolves this correctly — ``` GET /api/v1/repos/iterative-works/iw-cli/commits/811feec.../status → "state": "success", six contexts, each green ``` — but the per-status endpoint keeps the whole history: ``` GET /api/v1/repos/iterative-works/iw-cli/commits/811feec.../statuses → 38 entries, newest first: success CI / test 19:05:16 ← run #152 success CI / dashboard-build 18:57:50 ← run #152 ... failure CI / test 18:52:xx ← run #151, still in the list ``` `ci-gate` appears to scan all entries and treat any `failure` as the verdict, instead of taking the latest status per context (or asking the combined `/status` endpoint, which already does exactly this). ## Consequence Any close or phase gate that has ever had to re-run CI — for a flaky runner, a network blip, a cancelled run — is stuck reporting failure permanently, and writes a false `ci_failed` into the card the dashboard shows. The only way out is to push a new commit so the status history starts clean, or to correct the card by hand after checking the combined status. A second, smaller defect in the same run: the gate wrote `ci_gated_sha: e1cf757` — the commit *before* the merge it had just judged, not `811feec`. ## Suggested fix Resolve each context to its most recent status before deciding, or read the combined `/commits/{sha}/status` endpoint, whose `state` is already the answer. Stamp `ci_gated_sha` with the commit actually gated. ## Neighbours IW-437 covers the gate calling a *slow* pipeline failed (the wait budget expiring against a green run). This is the gate calling a *re-run* pipeline failed, which is a different cause with the same visible outcome — a green pull request marked `ci_failed`. Found while closing slice `dashboard-control-plane/08-the-cleanup-gets-done` (PR #470); recorded in that effort's `LESSONS.md`.

mph referenced this issue

2026-08-30 19:26:06 +00:00

phase-pr treats 'no pipeline yet' as 'no CI' and passes the gate before the pipeline is created #485

mph referenced this issue

2026-09-01 14:03:27 +00:00

phase-pr pushes a commit after reading CI, so its green can describe a stale head #491

mph referenced this issue from a commit

2026-09-02 06:22:12 +00:00

fix(ci-gate): let a re-triggered run supersede the one it replaced

mph referenced this issue

2026-09-02 06:22:57 +00:00

Squash the batch-implement blockers: nine fixes for the phase gate, the CI gate and the review-state card #501

mph commented

2026-09-02 20:55:14 +00:00

Author

Owner

Fixed in PR #501 (merge d1eba9c4). Run supersession now keys on (workflowLabel, event) and keeps the newest, after collapsing attempts per run id, so a re-triggered run supersedes the one it replaced. Runs tied at the newest instant are all kept. The second defect is fixed too: ci_gated_sha is stamped with the commit actually gated and cleared on every non-green exit.

Known limit, accepted deliberately: on GitHub workflowLabel is the run display name, so two workflow files sharing one name: and one event would collapse. Forgejo (workflow_id) and GitLab (one pipeline per commit+source) are unaffected.

Fixed in PR #501 (merge d1eba9c4). Run supersession now keys on `(workflowLabel, event)` and keeps the newest, after collapsing attempts per run id, so a re-triggered run supersedes the one it replaced. Runs tied at the newest instant are all kept. The second defect is fixed too: `ci_gated_sha` is stamped with the commit actually gated and cleared on every non-green exit. Known limit, accepted deliberately: on GitHub `workflowLabel` is the run display name, so two workflow files sharing one `name:` and one event would collapse. Forgejo (`workflow_id`) and GitLab (one pipeline per commit+source) are unaffected.

mph closed this issue

2026-09-02 20:55:14 +00:00