fix(ci): make the nightly contract suite runnable #412

Merged

mph merged 3 commits from fix/nightly-contract-suite into main 2026-07-30 19:03:50 +00:00

Conversation 0 Commits 3 Files changed 4 +28 -3

mph commented 2026-07-30 13:18:45 +00:00

Owner

Copy link

Summary

The nightly tool-contract job has failed on every one of its 34 scheduled runs, starting with the first one on 2026-06-27 — it has never been green. Two independent defects, both present since the suite landed in #377. Neither is related to recent work.

All other jobs (compile, format, lint, test, dashboard-build) were green throughout; only contract failed.

1. mill show output was parsed from a merged stream

mill_contract.bats parsed $output from a plain run, which merges stderr into it. Cold, dashboard.assembly builds the frontend, and build.mill:104-105 runs yarn and vite with stderr = os.Inherit — so jq received 29 lines of build chatter ahead of the jar ref and failed with Invalid numeric literal.

iw-run:51 reads stdout and diverts stderr to a file, so the merged stream was never the contract being pinned. Fixed by reading stdout alone via run --separate-stderr (with bats_require_minimum_version 1.5.0, or bats warns BW02).

This is why it was green locally and red in CI: a developer's out/ is warm, so the frontend never rebuilds and no stderr is emitted. Reproduced locally by deleting out/frontend/viteBuild.*.

iw-run itself was never affected — no production code changes here.

2. glab was never installed in the CI image

The suite probes glab's flag surface, but Dockerfile.ci installs gh, fj, JDK, scala-cli, node and mill — never glab. All five surface tests failed with exit 127. They passed locally only because a developer machine has glab installed.

Added as a pre-downloaded deps/ binary like the other major tools, rather than from a package repo at build time, matching the reproducibility the image already aims for. Verified glab 1.110.0 advertises every flag the suite asserts.

Verification

The rebuilt image is pushed (iw-cli-ci:latest → sha256:b23facad508e). The full contract suite was run inside it against a clean clone of this branch, cold — empty coursier cache, no out/:

• exit 0, all 35 tests pass (previously 10 failures)
• the frontend genuinely built during the run (viteBuild.log records Yarn 4.13.0 and vite 8.0.9), so the exact stderr that used to break the test was present and is now correctly separated
• scala-cli compile --scalac-option -Werror on core/ passed cold, so the coursier-cache-reset gotcha from the last image rebuild did not recur

This PR is labeled contract so the job runs here rather than waiting for the nightly.

Note

Worth deciding separately: 34 consecutive failures produced no signal anyone acted on. Even with both defects fixed, a drift detector nobody is notified about will rot again.

🤖 Generated with Claude Code

## Summary The nightly tool-contract job has failed on **every one of its 34 scheduled runs**, starting with the first one on 2026-06-27 — it has never been green. Two independent defects, both present since the suite landed in #377. Neither is related to recent work. All other jobs (compile, format, lint, test, dashboard-build) were green throughout; only `contract` failed. ## 1. `mill show` output was parsed from a merged stream `mill_contract.bats` parsed `$output` from a plain `run`, which merges stderr into it. Cold, `dashboard.assembly` builds the frontend, and `build.mill:104-105` runs yarn and vite with `stderr = os.Inherit` — so `jq` received 29 lines of build chatter ahead of the jar ref and failed with `Invalid numeric literal`. `iw-run:51` reads stdout and diverts stderr to a file, so the merged stream was never the contract being pinned. Fixed by reading stdout alone via `run --separate-stderr` (with `bats_require_minimum_version 1.5.0`, or bats warns BW02). This is why it was green locally and red in CI: a developer's `out/` is warm, so the frontend never rebuilds and no stderr is emitted. Reproduced locally by deleting `out/frontend/viteBuild.*`. `iw-run` itself was never affected — no production code changes here. ## 2. `glab` was never installed in the CI image The suite probes glab's flag surface, but `Dockerfile.ci` installs `gh`, `fj`, JDK, scala-cli, node and mill — never `glab`. All five surface tests failed with exit 127. They passed locally only because a developer machine has glab installed. Added as a pre-downloaded `deps/` binary like the other major tools, rather than from a package repo at build time, matching the reproducibility the image already aims for. Verified glab 1.110.0 advertises every flag the suite asserts. ## Verification The rebuilt image is pushed (`iw-cli-ci:latest` → `sha256:b23facad508e`). The full contract suite was run inside it against a clean clone of this branch, cold — empty coursier cache, no `out/`: - exit 0, **all 35 tests pass** (previously 10 failures) - the frontend genuinely built during the run (`viteBuild.log` records Yarn 4.13.0 and vite 8.0.9), so the exact stderr that used to break the test was present and is now correctly separated - `scala-cli compile --scalac-option -Werror` on `core/` passed cold, so the coursier-cache-reset gotcha from the last image rebuild did not recur This PR is labeled `contract` so the job runs here rather than waiting for the nightly. ## Note Worth deciding separately: 34 consecutive failures produced no signal anyone acted on. Even with both defects fixed, a drift detector nobody is notified about will rot again. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mph added 2 commits 2026-07-30 13:18:45 +00:00

fix(contract): pin mill show stdout, not stdout merged with stderr ... cee05bd910

The mill contract tests parsed `$output` from a plain `run`, which merges
stderr into it. Cold, `dashboard.assembly` builds the frontend and yarn and
vite both log to stderr, so `jq` got build chatter instead of the jar ref and
the test failed with "Invalid numeric literal".

`iw-run` reads stdout and diverts stderr to a file, so the merged stream was
never the contract. Read stdout alone via `run --separate-stderr`.

This is why the nightly contract job failed on every scheduled run: locally
`out/` is warm, so the frontend never rebuilds and no stderr is produced.

fix(ci): install glab in the CI image so the glab contract tests can run ... 94f2683af3

The tool-contract suite probes glab's flag surface, but glab was never in the
image, so its five surface tests failed with exit 127 on every scheduled run.
They passed locally only because a developer machine has glab installed.

Pre-downloaded into deps/ like the other major tools rather than installed from
a package repo at build time, matching the reproducibility the image already
aims for. Verified 1.110.0 advertises every flag the suite asserts.

mph added the

contract

label 2026-07-30 13:19:04 +00:00

mph added 1 commit 2026-07-30 13:20:49 +00:00

fix(ci): let the contract label actually trigger the contract job ... d04e2142b9

The contract job gates on `contains(pull_request.labels.*.name, 'contract')`,
but `pull_request` defaults to opened/synchronize/reopened. A label is normally
added after opening, so the run that evaluated the gate had already finished and
the job was skipped — the documented opt-in could not work.

Adding `labeled` to the trigger types re-runs the workflow when the label lands.
The label itself also did not exist in the repo and has been created.

mph merged commit d5efe3a184 into main 2026-07-30 19:03:50 +00:00

mph deleted branch fix/nightly-contract-suite 2026-07-30 19:03:50 +00:00

mph referenced this pull request from a commit 2026-07-30 19:03:51 +00:00

Merge pull request 'fix(ci): make the nightly contract suite runnable' (#412) from fix/nightly-contract-suite into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something isn't working

  

contract

Run the tool-contract suite on this PR

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

bug

contract

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

iterative-works/iw-cli!412

No description provided.