iterative-works/iw-cli

Watch

Fork 0

Hooks and CI read a crashed tool as a finding, and print --no-verify as the remedy #497

New issue

Closed

opened 2026-09-01 15:33:00 +00:00 by mph · 3 comments

mph commented

2026-09-01 15:33:00 +00:00

Owner

The deterministic fault

Five failure boxes assert a diagnosis they did not make, and all five offer the bypass:

.git-hooks/pre-commit:28 — COMMIT BLOCKED: Code formatting issues / Fix: scala-cli fmt .
.git-hooks/pre-commit:46 — COMMIT BLOCKED: Compilation errors/warnings
.git-hooks/pre-push:29, :47, :65 — the same shape for compile, scalafix and tests

Each is written as if <command>; then ... else <blocked box>; fi. The else branch cannot tell a finding from a crash, a timeout, or a killed process — but the text it prints commits to "finding", and the last line it prints is how to skip the gate. A transient tool crash therefore steers the developer straight past the check that would have caught a real fault next time.

Three crashes that produced a false diagnosis

2026-08-21, twice in a row, local pre-commit (recorded in IW-422 LESSONS.md):

ScalaNative: Unrecoverable NullPointerException in user thread
.git-hooks/pre-commit: line 32: Segmentation fault (core dumped)

The files were formatted. scala-cli fmt --check . run directly succeeded three times out of three.

2026-08-30, CI format job, PR 483 run 581, exitcode '139' (128 + 11 = SIGSEGV):

ScalaNative: Unrecoverable NullPointerException in user thread
Segmentation fault (core dumped) scala-cli fmt --check .

Its last progress line was out 0.0% 0, fmt 0.0% 0, in 0.0% 0 todo=512 — it had inspected none of the 512 files and named no unformatted file. The commit it ran on changed review-state.json only and contained no Scala. The same job passed on the preceding commit on the same image, and scala-cli fmt --check . passed locally over all 512 files.

2026-09-01, pre-push, the same shape in the test box: PUSH BLOCKED: Tests failed printed directly under 241/241, SUCCESS. Both steps exit 0 standalone and the hook passes when run directly; the identical push retried and succeeded. Tracked separately as part of #468.

Why this is filed now

IW-422 saw the formatter crash twice and deliberately did not file, because the crash is rare, not reproducible on demand, and upstream in the scala-cli ScalaNative launcher — pinning scala-cli would be a guess at the cause. That reasoning still holds for the crash itself. Its decision said the entry was held for recurrence. The CI sighting is that recurrence, and it removes the explanation IW-422 offered: a CI runner has neither the memory pressure nor the resident Mill daemons that entry blamed.

The reporting defect, unlike the crash, is deterministic and ours.

Wanted

A gate must distinguish a signal death from a finding. An exit code of 128 or more is a signal; scala-cli fmt also reports how many files it inspected, and zero inspected files with zero named faults is not a formatting failure.
On a crash, say the tool crashed and to retry. Do not print the bypass — the bypass is the wrong move for a transient crash and it is the only instruction the current box gives.
Keep --no-verify for the case where the tool really did object.

Recovery note

This Forgejo (16.x) has no rerun endpoint. Both POST /actions/jobs/<id>/rerun and POST /actions/runs/<id>/rerun answer 404, so re-triggering needs an empty commit. 45bca500 was that commit; run 582 then passed every job.

#465 — a failed push reports only 'failed to push some refs'; the hook's stderr is dropped.
#466 — ./iw ./test is green on code the pre-push hook rejects.
#443 — CI compile job must not fail on a Bloop server timeout. Same class, in CI.
#468 — carries the 2026-09-01 test-box sighting.

Recorded as IW-476 LESSONS.md entry 16 and IW-422 LESSONS.md entry on the scalafmt crash.

Every gate in this repository treats a non-zero exit as "the tool examined your code and objected". When the tool dies from a signal instead, that diagnosis is false, and the remedy the box prints is `--no-verify`. ## The deterministic fault Five failure boxes assert a diagnosis they did not make, and all five offer the bypass: - `.git-hooks/pre-commit:28` — `COMMIT BLOCKED: Code formatting issues` / `Fix: scala-cli fmt .` - `.git-hooks/pre-commit:46` — `COMMIT BLOCKED: Compilation errors/warnings` - `.git-hooks/pre-push:29`, `:47`, `:65` — the same shape for compile, scalafix and tests Each is written as `if <command>; then ... else <blocked box>; fi`. The `else` branch cannot tell a finding from a crash, a timeout, or a killed process — but the text it prints commits to "finding", and the last line it prints is how to skip the gate. A transient tool crash therefore steers the developer straight past the check that would have caught a real fault next time. ## Three crashes that produced a false diagnosis **2026-08-21, twice in a row, local `pre-commit`** (recorded in IW-422 LESSONS.md): ScalaNative: Unrecoverable NullPointerException in user thread .git-hooks/pre-commit: line 32: Segmentation fault (core dumped) The files were formatted. `scala-cli fmt --check .` run directly succeeded three times out of three. **2026-08-30, CI `format` job, PR 483 run 581**, `exitcode '139'` (128 + 11 = SIGSEGV): ScalaNative: Unrecoverable NullPointerException in user thread Segmentation fault (core dumped) scala-cli fmt --check . Its last progress line was `out 0.0% 0, fmt 0.0% 0, in 0.0% 0 todo=512` — it had inspected none of the 512 files and named no unformatted file. The commit it ran on changed `review-state.json` only and contained no Scala. The same job passed on the preceding commit on the same image, and `scala-cli fmt --check .` passed locally over all 512 files. **2026-09-01, `pre-push`**, the same shape in the test box: `PUSH BLOCKED: Tests failed` printed directly under `241/241, SUCCESS`. Both steps exit 0 standalone and the hook passes when run directly; the identical push retried and succeeded. Tracked separately as part of #468. ## Why this is filed now IW-422 saw the formatter crash twice and deliberately did not file, because the crash is rare, not reproducible on demand, and upstream in the scala-cli ScalaNative launcher — pinning scala-cli would be a guess at the cause. That reasoning still holds for the crash itself. Its decision said the entry was **held for recurrence**. The CI sighting is that recurrence, and it removes the explanation IW-422 offered: a CI runner has neither the memory pressure nor the resident Mill daemons that entry blamed. The reporting defect, unlike the crash, is deterministic and ours. ## Wanted - A gate must distinguish a signal death from a finding. An exit code of 128 or more is a signal; `scala-cli fmt` also reports how many files it inspected, and zero inspected files with zero named faults is not a formatting failure. - On a crash, say the tool crashed and to retry. Do not print the bypass — the bypass is the wrong move for a transient crash and it is the only instruction the current box gives. - Keep `--no-verify` for the case where the tool really did object. ## Recovery note This Forgejo (16.x) has no rerun endpoint. Both `POST /actions/jobs/<id>/rerun` and `POST /actions/runs/<id>/rerun` answer 404, so re-triggering needs an empty commit. `45bca500` was that commit; run 582 then passed every job. ## Related - #465 — a failed push reports only 'failed to push some refs'; the hook's stderr is dropped. - #466 — `./iw ./test` is green on code the pre-push hook rejects. - #443 — CI compile job must not fail on a Bloop server timeout. Same class, in CI. - #468 — carries the 2026-09-01 test-box sighting. Recorded as IW-476 LESSONS.md entry 16 and IW-422 LESSONS.md entry on the scalafmt crash.

mph referenced this issue from a commit

2026-09-01 15:35:00 +00:00

workflow: Triage lessons for IW-476

mph referenced this issue

2026-09-01 15:35:39 +00:00

IW-476: The herdr sidebar names a worktree by its key, and a rename sticks #498

mph referenced this issue

2026-09-01 19:46:01 +00:00

tmux-dependent BATS tests fail non-deterministically under a full ./iw ./test run #468

mph added the

needs-human

label

2026-09-05 00:43:09 +00:00

mph commented

2026-09-05 00:43:09 +00:00

Author

Owner

Intake: needs-human
Workflow: diagnostic
Reason: 2 open CLARIFY markers (G1) — whether to wrap the CI jobs, and what the crash box should say on a repeating crash
Analysis: project-management/issues/IW-497/analysis.md on branch IW-497

Intake: needs-human Workflow: diagnostic Reason: 2 open CLARIFY markers (G1) — whether to wrap the CI jobs, and what the crash box should say on a repeating crash Analysis: project-management/issues/IW-497/analysis.md on branch IW-497

claude referenced this issue

2026-09-21 12:05:33 +00:00

phase-pr pushes a commit after reading CI, so its green can describe a stale head #491

claude commented

2026-10-06 18:16:55 +00:00

Member

Intake: needs-input
Workflow: diagnostic
Reason: G1 (2 CLARIFY markers)
Analysis: project-management/issues/IW-497/analysis.md on branch IW-497

[G1] The fix makes the git hooks tell a crashed tool (killed by a signal, for example exit 139) from a real finding. CI prints no diagnosis of its own, so it makes no false statement, but a human who reads a red job can still read a crash as a finding (as nearly happened on run 581). Wrapping the format, lint, test, dashboard-build and contract steps of .github/workflows/ci.yml in the same helper would annotate a signal death in the job log, about 1-2 h more and a CI-visible change. The wrapper must never turn a non-zero status into a pass, and must keep each command string contiguous, because ScalafmtGateTest and ScalafixGateTest pin those strings. Should IW-497 wrap the CI jobs too? Options: a) yes, wrap those five CI steps in IW-497 b) no, change only the hooks now, and file the CI wrapping as its own issue c) no CI wrapping at all Recommended: a)
[G1] The compile job of CI (ci.yml:73-74) is also a candidate for the wrapper, but iw-cli #443 ("CI compile job must not fail on a Bloop server timeout", still open) changes the same lines. Where does the compile job's wrapper go? Options: a) in #443, so one issue edits those lines; IW-497 leaves the compile job alone b) in IW-497, and #443 builds on it c) nowhere for now Recommended: a)
[G1] The fix removes "Bypass: git commit --no-verify" from the hook's crash box (.git-hooks/pre-commit:40, 63, 81 print it today). For a transient crash, a retry works (the 2026-09-01 push passed on an identical retry). For a tool that crashes every time (a bad scala-cli install, a corrupt coursier cache), the developer then has no documented way out, and may find --no-verify without its warning. CONTRIBUTING.md:96-105 is the only bypass guidance; no policy exists for a gate whose tool is broken. What should the crash box say? Options: a) "Run the command yourself; if it passes directly and the hook keeps crashing, the tool is broken — fix the tool", and no bypass at all b) as a), and name the bypass on a repeated crash only, as "the gate could not run; you are committing unchecked code" — this needs a crash count stored in .git/ between runs c) keep the bypass line as it is today Recommended: a)

Answer in a comment on this issue; intake reads it on its next iteration.

Intake: needs-input Workflow: diagnostic Reason: G1 (2 CLARIFY markers) Analysis: project-management/issues/IW-497/analysis.md on branch IW-497 1. [G1] The fix makes the git hooks tell a crashed tool (killed by a signal, for example exit 139) from a real finding. CI prints no diagnosis of its own, so it makes no false statement, but a human who reads a red job can still read a crash as a finding (as nearly happened on run 581). Wrapping the `format`, `lint`, `test`, `dashboard-build` and `contract` steps of `.github/workflows/ci.yml` in the same helper would annotate a signal death in the job log, about 1-2 h more and a CI-visible change. The wrapper must never turn a non-zero status into a pass, and must keep each command string contiguous, because `ScalafmtGateTest` and `ScalafixGateTest` pin those strings. Should IW-497 wrap the CI jobs too? Options: a) yes, wrap those five CI steps in IW-497 b) no, change only the hooks now, and file the CI wrapping as its own issue c) no CI wrapping at all Recommended: a) 2. [G1] The `compile` job of CI (`ci.yml:73-74`) is also a candidate for the wrapper, but iw-cli #443 ("CI compile job must not fail on a Bloop server timeout", still open) changes the same lines. Where does the `compile` job's wrapper go? Options: a) in #443, so one issue edits those lines; IW-497 leaves the `compile` job alone b) in IW-497, and #443 builds on it c) nowhere for now Recommended: a) 3. [G1] The fix removes "Bypass: git commit --no-verify" from the hook's crash box (`.git-hooks/pre-commit:40, 63, 81` print it today). For a transient crash, a retry works (the 2026-09-01 push passed on an identical retry). For a tool that crashes every time (a bad scala-cli install, a corrupt coursier cache), the developer then has no documented way out, and may find `--no-verify` without its warning. `CONTRIBUTING.md:96-105` is the only bypass guidance; no policy exists for a gate whose tool is broken. What should the crash box say? Options: a) "Run the command yourself; if it passes directly and the hook keeps crashing, the tool is broken — fix the tool", and no bypass at all b) as a), and name the bypass on a repeated crash only, as "the gate could not run; you are committing unchecked code" — this needs a crash count stored in `.git/` between runs c) keep the bypass line as it is today Recommended: a) Answer in a comment on this issue; intake reads it on its next iteration.

claude added

needs-input

and removed

needs-human

labels

2026-10-06 18:16:55 +00:00

mph commented

2026-10-10 19:48:09 +00:00

Author

Owner

Drop it now

mph closed this issue