phase-merge: false CI timeout on a green PR — ci_deadline is inherited from an earlier gate, ci_gated_sha goes stale #433

New issue

Open

opened 2026-08-21 17:27:45 +00:00 by mph · 0 comments

mph commented

2026-08-21 17:27:45 +00:00

Owner

Problem

phase-merge can report a 30-minute CI timeout without waiting, exit 1, and write
status: ci_failed onto a card whose PR head has a green pipeline. Two independent defects
combine to produce it.

Defect A — `ci_deadline` is inherited from an earlier gate and never reset for a new head

CiGate.startOfWait (core/model/CiGate.scala:351-361) reconstructs the start of the wait from a
persisted deadline on the card:

deadline
  .flatMap(raw => try Some(java.time.Instant.parse(raw).toEpochMilli - timeoutMs) catch ...)
  .getOrElse(now)

now is used only when the field is absent or unparseable. So when an earlier phase-pr wrote
ci_deadline and then exited 3, a phase-merge run later inherits that same deadline — and if the
clock is already past it, the wait is born exhausted. The command reports
Timed out waiting for CI checks after 30m having waited seconds, regardless of what CI actually
says about the current head.

Defect B — `ci_gated_sha` is only written on a successful gate, so the card-only shortcut never engages

isStateOnlyDelta (core/commands/PhaseMerge.scala:144-149) exists precisely to stop a card-only
commit from costing a second CI cycle:

env.reviewState
  .readCiGatedSha(reviewPathFor(r, env))
  .flatMap(sha => env.git.diffNameOnly(sha, env.cwd).toOption)
  .exists(_.forall(_.endsWith("review-state.json")))

Its own docstring states the intent: "phase-pr gates the phase and then pushes its own card write,
which starts a run nobody is waiting for. Waiting for it doubles the CI a phase costs."

But ci_gated_sha is written only when a gate succeeds. When phase-pr keeps exiting 3, the
field is never refreshed, goes stale by whole phases, and diffNameOnly then returns the entire
phase changeset — so the mitigation can never fire, and every card write costs a full CI cycle.
Those cycles are what burn the deadline in Defect A.

Observed failure

Project medeca-modul-poptavky, issue MEDECA-407, phase 08, GitLab MR !407, 2026-08-21.

./iw phase-pr created the MR, waited, and exited 3:
Waited 501s of 30m. Nothing failed and nothing needs a person.
CI then went green: pipeline 5169 on 443d4a3aa — success.
Card state was moved to awaiting_review and pushed. Pipeline 5170 on the new head
4214ca936 (a review-state.json-only commit) — also success.
./iw phase-merge --max-retries 0 returned in well under its own 30-minute budget with:
```
Error: Timed out waiting for CI checks after 30m.
Error: PR is at .../merge_requests/407. You can merge manually once CI passes.
```
exit 1, and left behind an unpushed commit plus a working-tree edit that between them wrote
status: ci_pending → ci_failed.
Merging by hand succeeded immediately: glab mr merge 407 --squash printed
✓ Pipeline succeeded. ✓ Merged!

The card fields at the time

Field	Value	Meaning
`ci_deadline`	`2026-08-21T16:49:31.994Z`	~90 minutes in the past when `phase-merge` ran (~18:2x)
`ci_gated_sha`	`411d6fe68`	the commit "chore(MEDECA-407): update review-state for phase 05"

git log -S'ci_gated_sha' on the card shows the value last changed back at phase 02 — it had
been stale for six phases, so isStateOnlyDelta had been dead that whole time.

With startOfWait computing startTime = 16:49:31 − 30m = 16:19:31, the wait was already ~2 hours
old at birth. Hence the instant "timed out after 30m".

Impact

A green PR is reported as a CI failure. The card says ci_failed; the forge says the pipeline
succeeded.
An agent cannot converge. The documented recovery is to run the command again, but each
invocation pushes another card commit, starting another pipeline, while the inherited deadline
stays expired. The consuming project recorded this livelock twice — once for phase-pr in phase 07
(five pipelines, 5161-5165, four of which reported success while the command reported a timeout),
and again for phase-merge in phase 08.
A human has to verify by hand which pipeline belongs to the PR head and merge outside the tool,
which is exactly what the gate exists to avoid.

Suggested directions

Offered as starting points, not a prescription:

Reset or re-key ci_deadline when the head SHA under gate changes — a new commit is a new
pipeline and deserves a fresh budget. Storing the wait's start keyed by head SHA, rather than an
absolute deadline, would make this structural.
Write ci_gated_sha whenever a gate observes green for a SHA, not only on the path that also
merges and advances. That keeps the card-only shortcut alive across exit 3 waits, which is when
it is most needed.
Never report "timed out after N" when less than N has actually elapsed in this invocation. If
an inherited budget is already spent, re-read CI once and report the real verdict instead of a
timeout.
Consider not pushing a card commit ahead of the gate (CommitPolicy.CommitBeforeGate), or
arranging that review-state.json-only commits do not trigger a pipeline at all. The cheapest fix
for "the retry invalidates the run it is waiting for" is to stop creating the extra run.

#418 — introduced the CI gate and ci_deadline.
#400 — adjacent but distinct: batch-implement spending its retry budget on ci_pending. This
issue is about the deadline and gated-sha mechanism inside phase-pr/phase-merge themselves.

## Problem `phase-merge` can report a 30-minute CI timeout **without waiting**, exit `1`, and write `status: ci_failed` onto a card whose PR head has a **green** pipeline. Two independent defects combine to produce it. ### Defect A — `ci_deadline` is inherited from an earlier gate and never reset for a new head `CiGate.startOfWait` (`core/model/CiGate.scala:351-361`) reconstructs the start of the wait from a **persisted** deadline on the card: ```scala deadline .flatMap(raw => try Some(java.time.Instant.parse(raw).toEpochMilli - timeoutMs) catch ...) .getOrElse(now) ``` `now` is used only when the field is absent or unparseable. So when an earlier `phase-pr` wrote `ci_deadline` and then exited 3, a `phase-merge` run later inherits that same deadline — and if the clock is already past it, the wait is **born exhausted**. The command reports `Timed out waiting for CI checks after 30m` having waited seconds, regardless of what CI actually says about the current head. ### Defect B — `ci_gated_sha` is only written on a successful gate, so the card-only shortcut never engages `isStateOnlyDelta` (`core/commands/PhaseMerge.scala:144-149`) exists precisely to stop a card-only commit from costing a second CI cycle: ```scala env.reviewState .readCiGatedSha(reviewPathFor(r, env)) .flatMap(sha => env.git.diffNameOnly(sha, env.cwd).toOption) .exists(_.forall(_.endsWith("review-state.json"))) ``` Its own docstring states the intent: *"`phase-pr` gates the phase and then pushes its own card write, which starts a run nobody is waiting for. Waiting for it doubles the CI a phase costs."* But `ci_gated_sha` is written only when a gate **succeeds**. When `phase-pr` keeps exiting 3, the field is never refreshed, goes stale by whole phases, and `diffNameOnly` then returns the entire phase changeset — so the mitigation can never fire, and every card write costs a full CI cycle. Those cycles are what burn the deadline in Defect A. ## Observed failure Project `medeca-modul-poptavky`, issue MEDECA-407, phase 08, GitLab MR !407, 2026-08-21. 1. `./iw phase-pr` created the MR, waited, and exited **3**: `Waited 501s of 30m. Nothing failed and nothing needs a person.` 2. CI then went **green**: pipeline 5169 on `443d4a3aa` — `success`. 3. Card state was moved to `awaiting_review` and pushed. Pipeline **5170** on the new head `4214ca936` (a `review-state.json`-only commit) — also **success**. 4. `./iw phase-merge --max-retries 0` returned in well under its own 30-minute budget with: ``` Error: Timed out waiting for CI checks after 30m. Error: PR is at .../merge_requests/407. You can merge manually once CI passes. ``` exit `1`, and left behind an unpushed commit plus a working-tree edit that between them wrote `status: ci_pending` → `ci_failed`. 5. Merging by hand succeeded immediately: `glab mr merge 407 --squash` printed `✓ Pipeline succeeded. ✓ Merged!` ### The card fields at the time | Field | Value | Meaning | |---|---|---| | `ci_deadline` | `2026-08-21T16:49:31.994Z` | ~90 minutes **in the past** when `phase-merge` ran (~18:2x) | | `ci_gated_sha` | `411d6fe68` | the commit *"chore(MEDECA-407): update review-state for **phase 05**"* | `git log -S'ci_gated_sha'` on the card shows the value last changed back at **phase 02** — it had been stale for six phases, so `isStateOnlyDelta` had been dead that whole time. With `startOfWait` computing `startTime = 16:49:31 − 30m = 16:19:31`, the wait was already ~2 hours old at birth. Hence the instant "timed out after 30m". ## Impact - **A green PR is reported as a CI failure.** The card says `ci_failed`; the forge says the pipeline succeeded. - **An agent cannot converge.** The documented recovery is to run the command again, but each invocation pushes another card commit, starting another pipeline, while the inherited deadline stays expired. The consuming project recorded this livelock twice — once for `phase-pr` in phase 07 (five pipelines, 5161-5165, four of which reported `success` while the command reported a timeout), and again for `phase-merge` in phase 08. - **A human has to verify by hand** which pipeline belongs to the PR head and merge outside the tool, which is exactly what the gate exists to avoid. ## Suggested directions Offered as starting points, not a prescription: 1. **Reset or re-key `ci_deadline` when the head SHA under gate changes** — a new commit is a new pipeline and deserves a fresh budget. Storing the wait's *start* keyed by head SHA, rather than an absolute deadline, would make this structural. 2. **Write `ci_gated_sha` whenever a gate observes green for a SHA**, not only on the path that also merges and advances. That keeps the card-only shortcut alive across `exit 3` waits, which is when it is most needed. 3. **Never report "timed out after N" when less than N has actually elapsed in this invocation.** If an inherited budget is already spent, re-read CI once and report the real verdict instead of a timeout. 4. **Consider not pushing a card commit ahead of the gate** (`CommitPolicy.CommitBeforeGate`), or arranging that `review-state.json`-only commits do not trigger a pipeline at all. The cheapest fix for "the retry invalidates the run it is waiting for" is to stop creating the extra run. ## Related - #418 — introduced the CI gate and `ci_deadline`. - #400 — adjacent but distinct: `batch-implement` spending its *retry budget* on `ci_pending`. This issue is about the *deadline* and *gated-sha* mechanism inside `phase-pr`/`phase-merge` themselves.