iterative-works/iw-cli

Watch

Fork 0

CI compile job must not fail on a Bloop server timeout #443

New issue

Open

opened 2026-08-23 19:44:11 +00:00 by mph · 2 comments

mph commented

2026-08-23 19:44:11 +00:00

Owner

The CI compile job fails on a build server that never starts, and every layer above it reports the failure as a code defect.

Evidence

Opening PR 435 for IW-423, ./iw phase-pr reported CI failed — 1 job(s): compile (failure) and pointed at .iw/ci-logs/124-compile.log.

The log contains no compile error at all — rg -n '\[error\]|error:|FAILED' over its 1956 lines matches nothing. What it contains is:

17:58:30 Starting compilation server
18:00:04 Exception in thread "main" java.util.concurrent.TimeoutException: Future timed out after [30 seconds]

The compiler never ran. The same command the job runs, scala-cli compile --scalac-option -Werror core/, exits 0 locally on the same tree, and the pre-push hook had already run it clean when the branch was pushed. CI was re-run and went green (run 486) with no code change.

The same failure locally

This is the CI-container instance of a failure also seen on a developer host during IW-423: a Bloop daemon up 3h at 25.1 GB RSS against -Xmx8g (scala-cli's default BLOOP_JAVA_OPTS, which overrides the -XX:MaxRAMPercentage=10 also present in its environment), with java.lang.OutOfMemoryError: Java heap space repeated in ~/.local/share/scalacli/bloop/daemon/output. scala-cli bloop exit and a retry compiled clean.

Third sighting on this toolchain of a long-lived build daemon failing as something other than memory (the first was Mill's worker: Worker wire broken, worker likely crashed).

Lesson: a CI job whose build server fails to start reports the job name, and a phase gate that names the job reports "CI failed" — neither statement distinguishes a compiler that found errors from a compiler that never ran, so the first move on a red compile job is to grep the log for an actual error before believing it.

Fix

Either, or both:

Give the CI container's Bloop a bounded heap that matches the runner.
Have the compile step retry once after scala-cli bloop exit.

Note that phase-pr's own failure report is already good here: it printed the log path and the two rg queries that answered the question in one step. The developer-side half — .git-hooks/pre-push misreporting the same failure as a compilation error — is being addressed in the IW-423 branch.

Recorded during IW-423; see project-management/issues/IW-423/LESSONS.md entries 15 and 20.

The CI `compile` job fails on a build server that never starts, and every layer above it reports the failure as a code defect. ## Evidence Opening PR 435 for IW-423, `./iw phase-pr` reported `CI failed — 1 job(s): compile (failure)` and pointed at `.iw/ci-logs/124-compile.log`. The log contains **no compile error at all** — `rg -n '\[error\]|error:|FAILED'` over its 1956 lines matches nothing. What it contains is: ``` 17:58:30 Starting compilation server 18:00:04 Exception in thread "main" java.util.concurrent.TimeoutException: Future timed out after [30 seconds] ``` The compiler never ran. The same command the job runs, `scala-cli compile --scalac-option -Werror core/`, exits 0 locally on the same tree, and the `pre-push` hook had already run it clean when the branch was pushed. CI was re-run and went green (run 486) with no code change. ## The same failure locally This is the CI-container instance of a failure also seen on a developer host during IW-423: a Bloop daemon up 3h at 25.1 GB RSS against `-Xmx8g` (scala-cli's default `BLOOP_JAVA_OPTS`, which overrides the `-XX:MaxRAMPercentage=10` also present in its environment), with `java.lang.OutOfMemoryError: Java heap space` repeated in `~/.local/share/scalacli/bloop/daemon/output`. `scala-cli bloop exit` and a retry compiled clean. Third sighting on this toolchain of a long-lived build daemon failing as something other than memory (the first was Mill's worker: `Worker wire broken, worker likely crashed`). **Lesson:** a CI job whose build server fails to start reports the job name, and a phase gate that names the job reports "CI failed" — neither statement distinguishes a compiler that found errors from a compiler that never ran, so the first move on a red `compile` job is to grep the log for an actual error before believing it. ## Fix Either, or both: - Give the CI container's Bloop a bounded heap that matches the runner. - Have the compile step retry once after `scala-cli bloop exit`. Note that `phase-pr`'s own failure report is already good here: it printed the log path and the two `rg` queries that answered the question in one step. The developer-side half — `.git-hooks/pre-push` misreporting the same failure as a compilation error — is being addressed in the IW-423 branch. --- Recorded during IW-423; see `project-management/issues/IW-423/LESSONS.md` entries 15 and 20.

mph referenced this issue from a commit

2026-08-23 20:00:59 +00:00

workflow: Triage lessons for IW-423

mph referenced this issue

2026-09-01 12:07:50 +00:00

E2E and dashboard suites flake when another worktree builds concurrently #489

mph referenced this issue

2026-09-01 15:32:30 +00:00

JDK_JAVA_OPTIONS in the environment breaks Mill's daemon RPC, reported as 'Worker wire broken' #494

mph referenced this issue

2026-09-01 15:33:00 +00:00

Hooks and CI read a crashed tool as a finding, and print --no-verify as the remedy #497

mph commented

2026-09-01 15:33:16 +00:00

Author

Owner

The local instance of this, seen during IW-436 (2026-08-30), pushing the IW-436 feature branch after Phase 2 merged.

git push was blocked with PUSH BLOCKED: Tests failed and this output:

Error:   ✗ config.scala - compilation failed
Error: java.util.concurrent.TimeoutException
Error:   ✗ dashboard.scala - compilation failed
Bloop 'bsp' command exited with code 1. Something may be wrong with the
current configuration.
Error:   ✗ doctor.scala - compilation failed
Error: 3 of 34 commands failed to compile

No source file had changed. An immediate ./iw ./test compile re-run passed all 34 commands, and the following git push passed every pre-push check. The three failures were one Bloop BSP server timeout, not a compile error.

So the same misreport this issue records for CI happens at the pre-push gate: the command-compilation step counts any non-zero scala-cli exit as compilation failed, so an infrastructure timeout is indistinguishable from a real error in the summary line.

Wanted here too: separate a scala-cli/Bloop infrastructure failure — TimeoutException, Bloop 'bsp' command exited — from a genuine compile error, and retry once before failing the push.

Why the push case is worse than the CI case: the standard reaction to a blocked push is --no-verify, which our rules forbid and which would then also skip the checks that do matter.

On this issue's open question

This issue notes "the first was Mill's worker: Worker wire broken, worker likely crashed" without finding its cause. That cause is now found and filed as #494 — JDK_JAVA_OPTIONS in the environment makes every spawned JVM print a NOTE: line onto the pipe Mill's daemon parses as JSON. It is a different mechanism from the Bloop timeout, so the two should not be merged, but it does close that aside.

Recorded during IW-436; see project-management/issues/IW-436/LESSONS.md entry 8.

The local instance of this, seen during IW-436 (2026-08-30), pushing the `IW-436` feature branch after Phase 2 merged. `git push` was blocked with `PUSH BLOCKED: Tests failed` and this output: ``` Error: ✗ config.scala - compilation failed Error: java.util.concurrent.TimeoutException Error: ✗ dashboard.scala - compilation failed Bloop 'bsp' command exited with code 1. Something may be wrong with the current configuration. Error: ✗ doctor.scala - compilation failed Error: 3 of 34 commands failed to compile ``` No source file had changed. An immediate `./iw ./test compile` re-run passed all 34 commands, and the following `git push` passed every pre-push check. The three failures were **one Bloop BSP server timeout**, not a compile error. So the same misreport this issue records for CI happens at the pre-push gate: the command-compilation step counts any non-zero scala-cli exit as `compilation failed`, so an infrastructure timeout is indistinguishable from a real error in the summary line. **Wanted here too:** separate a scala-cli/Bloop infrastructure failure — `TimeoutException`, `Bloop 'bsp' command exited` — from a genuine compile error, and retry once before failing the push. **Why the push case is worse than the CI case:** the standard reaction to a blocked push is `--no-verify`, which our rules forbid and which would then also skip the checks that do matter. ## On this issue's open question This issue notes "the first was Mill's worker: `Worker wire broken, worker likely crashed`" without finding its cause. That cause is now found and filed as #494 — `JDK_JAVA_OPTIONS` in the environment makes every spawned JVM print a `NOTE:` line onto the pipe Mill's daemon parses as JSON. It is a different mechanism from the Bloop timeout, so the two should not be merged, but it does close that aside. --- Recorded during IW-436; see `project-management/issues/IW-436/LESSONS.md` entry 8.

mph referenced this issue from a commit

2026-09-01 15:36:36 +00:00

workflow: Triage lessons for IW-436

mph referenced this issue

2026-09-01 15:36:57 +00:00

IW-436: review-state CLI — fix five flag-parsing and update-semantics defects #499

mph referenced this issue

2026-09-02 06:22:57 +00:00

Squash the batch-implement blockers: nine fixes for the phase gate, the CI gate and the review-state card #501

claude referenced this issue

2026-10-06 18:16:55 +00:00

Hooks and CI read a crashed tool as a finding, and print --no-verify as the remedy #497

claude commented

2026-10-16 02:26:01 +00:00

Member

Intake: needs-input
Workflow: diagnostic
Reason: G7 (1 [human] finding)
Analysis: project-management/issues/IW-443/analysis.md on branch IW-443

1. Should the pre-commit hook also get the stop-and-retry on a build-server timeout, or stay as it is? (G7)

The analysis sends the core compile in both the pre-commit hook and pre-push step 1 through the new scripts/scala-cli-compile.sh. On a build-server timeout, that script stops the Bloop daemon with SIGKILL and retries once. The final review found that neither the issue body nor the comment asks for a pre-commit change. The comment asks only to "retry once before failing the push", and pre-commit is not needed for the push recovery. No recorded failure happened at pre-commit.

.git-hooks/pre-commit:71 and .git-hooks/pre-push:53 (step 1) call the script in place of scala-cli compile.

Links: analysis.md#L485, #L56, #L579, pre-commit#L69-L71, comment

a) Pre-commit also uses the script. A frozen daemon no longer blocks a commit, and both hooks behave the same. This adds one line in the hook and one wiring-test check. It goes past what the issue asks.
b) Pre-commit stays as it is. Only the pre-push hook and the command compile check change. The scope stays as the issue asks. A frozen daemon still blocks a commit with "COMMIT BLOCKED", and --no-verify is not permitted, so someone must stop the daemon by hand.

Recommended: a): The failure and the daemon are the same, the cost is one line, and with b) a commit can stay blocked with no permitted bypass.

Answer in a comment on this issue, per number, for example "1: a", or in your own words; intake reads it on its next iteration.

Intake: needs-input Workflow: diagnostic Reason: G7 (1 [human] finding) Analysis: project-management/issues/IW-443/analysis.md on branch IW-443 ### 1. Should the pre-commit hook also get the stop-and-retry on a build-server timeout, or stay as it is? (G7) The analysis sends the core compile in both the pre-commit hook and pre-push step 1 through the new `scripts/scala-cli-compile.sh`. On a build-server timeout, that script stops the Bloop daemon with SIGKILL and retries once. The final review found that neither the issue body nor the comment asks for a pre-commit change. The comment asks only to "retry once before failing the push", and pre-commit is not needed for the push recovery. No recorded failure happened at pre-commit. > `.git-hooks/pre-commit:71` and `.git-hooks/pre-push:53` (step 1) call the script in place of `scala-cli compile`. Links: [analysis.md#L485](https://code.iterative.works/iterative-works/iw-cli/src/branch/IW-443/project-management/issues/IW-443/analysis.md#L485-L485), [#L56](https://code.iterative.works/iterative-works/iw-cli/src/branch/IW-443/project-management/issues/IW-443/analysis.md#L56-L56), [#L579](https://code.iterative.works/iterative-works/iw-cli/src/branch/IW-443/project-management/issues/IW-443/analysis.md#L579-L579), [pre-commit#L69-L71](https://code.iterative.works/iterative-works/iw-cli/src/branch/IW-443/.git-hooks/pre-commit#L69-L71), [comment](https://code.iterative.works/iterative-works/iw-cli/issues/443#issuecomment-1383) - **a) Pre-commit also uses the script.** A frozen daemon no longer blocks a commit, and both hooks behave the same. This adds one line in the hook and one wiring-test check. It goes past what the issue asks. - **b) Pre-commit stays as it is.** Only the pre-push hook and the command compile check change. The scope stays as the issue asks. A frozen daemon still blocks a commit with "COMMIT BLOCKED", and `--no-verify` is not permitted, so someone must stop the daemon by hand. **Recommended: a)**: The failure and the daemon are the same, the cost is one line, and with b) a commit can stay blocked with no permitted bypass. Answer in a comment on this issue, per number, for example "1: a", or in your own words; intake reads it on its next iteration.

claude added the

needs-input

label

2026-10-16 02:26:01 +00:00