phase-pr re-triggers CI on every invocation, so a slow build can never be observed green #445

New issue

mph · 2026-08-25T11:58:17Z

mph commented

2026-08-25 11:58:17 +00:00

Each iw phase-pr invocation commits a review-state update (chore(<ISSUE>): update review-state for phase NN) and pushes it. On GitLab that push starts a new merge-request pipeline, and the gate then waits on the newest one. When one build takes longer than one poll window the loop cannot converge: the wait expires, the documented recovery is "run the same command again", that run pushes another commit, another pipeline starts, and the gate waits on a build that has just begun.

Observed

MR !410 on CMI/mdr/medeca-modul-poptavky (GitLab, gitlab.e-bs.cz). Build duration 668s against a ~510s poll window produced five pipelines on one branch — 5176, 5177, 5178 (all success), 5179 (running), 5180 (pending) — from four invocations, over four review-state commits (31542745c, bdf9af575, c9adce409, 19598688b) stacked on one real code commit (a83fbbfe7).

The 30m cumulative budget then expired and the gate reported Error: CI timed out, leaving review-state at ci_failed while three builds of the same tree had already passed.

Why it misleads twice

ci_failed reads as "CI refused the phase", and the skill's own recovery text says to read the named failing jobs — but no job failed, and there are none to name.
Raising --timeout does not help: the budget is measured cumulatively from the first wait, so --timeout 90m on a later invocation still expired at the original 30m mark.

Diagnosis without pushing anything further

glab api "projects/:id/pipelines?per_page=8"

lists every pipeline on the branch with its status, which makes the churn and the earlier green runs visible at once.

Proposed fix

Either do not push the review-state commit until CI has been observed, or pin the wait to the pipeline id recorded when the PR was created rather than to "the newest pipeline on the branch".

Found during MEDECA-430 (medeca-modul-poptavky); recorded in that issue's LESSONS.md entry 14 and promoted at triage.

Each `iw phase-pr` invocation commits a review-state update (`chore(<ISSUE>): update review-state for phase NN`) and pushes it. On GitLab that push starts a **new** merge-request pipeline, and the gate then waits on the newest one. When one build takes longer than one poll window the loop cannot converge: the wait expires, the documented recovery is "run the same command again", that run pushes another commit, another pipeline starts, and the gate waits on a build that has just begun. ## Observed MR !410 on `CMI/mdr/medeca-modul-poptavky` (GitLab, gitlab.e-bs.cz). Build duration 668s against a ~510s poll window produced five pipelines on one branch — 5176, 5177, 5178 (all `success`), 5179 (`running`), 5180 (`pending`) — from four invocations, over four review-state commits (`31542745c`, `bdf9af575`, `c9adce409`, `19598688b`) stacked on one real code commit (`a83fbbfe7`). The 30m cumulative budget then expired and the gate reported `Error: CI timed out`, leaving review-state at `ci_failed` while three builds of the same tree had already passed. ## Why it misleads twice - `ci_failed` reads as "CI refused the phase", and the skill's own recovery text says to read the named failing jobs — but no job failed, and there are none to name. - Raising `--timeout` does not help: the budget is measured cumulatively from the first wait, so `--timeout 90m` on a later invocation still expired at the original 30m mark. ## Diagnosis without pushing anything further ``` glab api "projects/:id/pipelines?per_page=8" ``` lists every pipeline on the branch with its status, which makes the churn and the earlier green runs visible at once. ## Proposed fix Either do not push the review-state commit until CI has been observed, or pin the wait to the pipeline id recorded when the PR was created rather than to "the newest pipeline on the branch". --- Found during MEDECA-430 (medeca-modul-poptavky); recorded in that issue's `LESSONS.md` entry 14 and promoted at triage.

mph commented

2026-08-26 09:44:37 +00:00

Additional observation from MEDECA-378 (same project, GitLab gitlab.e-bs.cz), phases 02 and 03.

On GitLab merge-result pipelines the re-triggered pipeline does not merely queue alongside the one being waited on — it cancels it. glab ci list for phase 03:

#5211 success   5b85bc725   <- the real phase commit
#5212 cancelled
#5213 cancelled
#5215 cancelled
#5216 cancelled
#5219 cancelled
#5220 pending

Every commit after 5b85bc725 differs from the previous one only by review-state.json. The gate then reported "CI timed out" after 30 minutes without a single job having started, while the phase commit itself had already passed in #5211.

So the failure mode is worse than "waits on a newer build": the older, green build is actively destroyed by the resume, which is why reading the pipeline list after the fact is the only way to discover the phase was green all along.

Confirms the cumulative-budget note in the original report: --timeout 90m on a resumed wait did not extend the already-recorded budget.

Additional observation from MEDECA-378 (same project, GitLab gitlab.e-bs.cz), phases 02 and 03. On GitLab merge-result pipelines the re-triggered pipeline does not merely queue alongside the one being waited on — it **cancels** it. `glab ci list` for phase 03: ``` #5211 success 5b85bc725 <- the real phase commit #5212 cancelled #5213 cancelled #5215 cancelled #5216 cancelled #5219 cancelled #5220 pending ``` Every commit after `5b85bc725` differs from the previous one only by `review-state.json`. The gate then reported "CI timed out" after 30 minutes without a single job having started, while the phase commit itself had already passed in #5211. So the failure mode is worse than "waits on a newer build": the older, green build is actively destroyed by the resume, which is why reading the pipeline list after the fact is the only way to discover the phase was green all along. Confirms the cumulative-budget note in the original report: `--timeout 90m` on a resumed wait did not extend the already-recorded budget.

mph referenced this issue from a pull request that will close it,

2026-08-28 07:22:08 +00:00

IW-437: The phase CI gate finishes a phase without manual intervention #475

mph closed this issue

2026-08-28 07:55:46 +00:00

mph referenced this issue

2026-08-30 19:26:06 +00:00

phase-pr treats 'no pipeline yet' as 'no CI' and passes the gate before the pipeline is created #485

Sign in to join this conversation.