iterative-works/iw-cli

Watch

Fork 0

format job crashes the formatter instead of checking formatting #512

New issue

Closed

opened 2026-09-03 08:12:31 +00:00 by mph · 4 comments

mph commented

2026-09-03 08:12:31 +00:00

Owner

scala-cli fmt --check . died in the format job with a Scala Native crash, exit 139:

Downloading https://github.com/scalameta/scalafmt/releases/download/v3.10.4/scalafmt-x86_64-pc-linux.zip
Downloaded ...
Looking for unformatted files...
  out 0.0% 0, fmt 0.0% 0, in 0.0% 0 todo=523
ScalaNative: Unrecoverable NullPointerException in user thread
/var/run/act/workflow/1.sh: line 2:   162 Segmentation fault      (core dumped) scala-cli fmt --check .

Run 630 on PR #511, job 1840. It crashed at 0% — before formatting anything — so it is not a file the formatter cannot parse.

Not reproducible locally

Same command, same .scalafmt.conf (version = "3.10.4"), same 523 files: exits 0 with "All files are formatted with scalafmt :)". The job succeeded on runs 625 and 627 within the preceding hour, on the same branch family.

Why it matters

format is a required check, so a crash in the tool blocks a merge with a message that reads like a formatting failure. The banner says the code is unformatted; the code is formatted. That is the same class of defect as #468's "PUSH BLOCKED over passing output" — the gate reporting something other than what it found.

Worth checking

The format and lint jobs run in parallel on one self-hosted runner and both fetch tools into a shared coursier cache. Contention there is the first thing to rule out.
Container memory: a native binary that is OOM-killed can surface as a SIGSEGV rather than as an OOM message.
It crashed in the same run in which actions/checkout@v4 had been rate-limited by GitHub twice, so the host may simply have been under pressure.

format also downloads scalafmt from github.com unauthenticated, which is the dependency that took the whole pipeline down twice (fixed for actions in 9559b0f9, still present for the formatter binary). A rate limit there would fail this job too, with a different message.

`scala-cli fmt --check .` died in the `format` job with a Scala Native crash, exit 139: ``` Downloading https://github.com/scalameta/scalafmt/releases/download/v3.10.4/scalafmt-x86_64-pc-linux.zip Downloaded ... Looking for unformatted files... out 0.0% 0, fmt 0.0% 0, in 0.0% 0 todo=523 ScalaNative: Unrecoverable NullPointerException in user thread /var/run/act/workflow/1.sh: line 2: 162 Segmentation fault (core dumped) scala-cli fmt --check . ``` Run 630 on PR #511, job 1840. It crashed at 0% — before formatting anything — so it is not a file the formatter cannot parse. ## Not reproducible locally Same command, same `.scalafmt.conf` (`version = "3.10.4"`), same 523 files: exits 0 with "All files are formatted with scalafmt :)". The job succeeded on runs 625 and 627 within the preceding hour, on the same branch family. ## Why it matters `format` is a required check, so a crash in the tool blocks a merge with a message that reads like a formatting failure. The banner says the code is unformatted; the code is formatted. That is the same class of defect as #468's "PUSH BLOCKED over passing output" — the gate reporting something other than what it found. ## Worth checking - The `format` and `lint` jobs run in parallel on one self-hosted runner and both fetch tools into a shared coursier cache. Contention there is the first thing to rule out. - Container memory: a native binary that is OOM-killed can surface as a SIGSEGV rather than as an OOM message. - It crashed in the same run in which `actions/checkout@v4` had been rate-limited by GitHub twice, so the host may simply have been under pressure. ## Related `format` also downloads scalafmt from `github.com` unauthenticated, which is the dependency that took the whole pipeline down twice (fixed for actions in 9559b0f9, still present for the formatter binary). A rate limit there would fail this job too, with a different message.

mph commented

2026-09-03 08:33:02 +00:00

Author

Owner

Did not reproduce on the next run of the same branch (9559b0f9): format passed, and so did every other job. Same 523 files, same scalafmt 3.10.4, same runner.

So it is intermittent, not a file the formatter cannot parse — which the 0% progress already suggested. It stays open: an intermittent crash in a required check still blocks a merge while reporting something other than what it found, and the shared-cache and memory questions above are unanswered.

Did not reproduce on the next run of the same branch (`9559b0f9`): `format` passed, and so did every other job. Same 523 files, same scalafmt 3.10.4, same runner. So it is intermittent, not a file the formatter cannot parse — which the 0% progress already suggested. It stays open: an intermittent crash in a required check still blocks a merge while reporting something other than what it found, and the shared-cache and memory questions above are unanswered.

mph referenced this issue

2026-09-03 08:33:43 +00:00

Make the forge the authority on a branch's pull request, and stop the test suites lying about themselves #511

mph added the

needs-human

label

2026-09-05 00:06:44 +00:00

mph commented

2026-09-05 00:06:44 +00:00

Author

Owner

Intake: needs-human
Workflow: diagnostic
Reason: 3 open CLARIFY markers (G1) — all three need access to the self-hosted runner host, which the worktree cannot reach
Analysis: project-management/issues/IW-512/analysis.md on branch IW-512

Intake: needs-human Workflow: diagnostic Reason: 3 open CLARIFY markers (G1) — all three need access to the self-hosted runner host, which the worktree cannot reach Analysis: project-management/issues/IW-512/analysis.md on branch IW-512

claude commented

2026-10-06 18:16:55 +00:00

Member

[G1] The format job crashed with exit 139 in 2 of 79 runs. The fix for Hypothesis 1 (a scalafmt version whose native binary does not crash; .scalafmt.conf pins 3.10.4) does not wait for these answers. Hypotheses 2-4 turn on the runner's container setup, which is in the Forgejo runner's config.yml on the host, not in this repository: memory.max and cpu.max of a job container, whether nproc inside it reports the host's cores, whether a host directory is mounted as the coursier cache, and the runner's capacity (job containers at once). What do you give? Options: a) the four values, in the answer b) the values are not known now: build the Hypothesis 1 fix and the crash instrumentation of Defect 2, which makes the next crash report these facts in the job log, and leave Hypotheses 2-4 open Recommended: b)
[G1] Only the host can confirm the cause of the crash of 2026-09-03 07:59:09Z (and 2026-08-30 10:34:49Z): an OOM kill in dmesg -T near those times, or the core file of "Segmentation fault (core dumped)" (its place is kernel.core_pattern), whose backtrace names the faulting frame. The logs show exit 139 and no OOM message, which already argues against an OOM kill (exit 137) and Scala Native heap exhaustion (exit 1). What do you give? Options: a) what dmesg and the core file show, in the answer b) the evidence is gone: rely on the signature match and the Defect 2 instrumentation Recommended: b)
[G1] Whether the two crashes correlate with load decides only whether the analysis adds a parallelism cap (Hypothesis 2) beside the version bump; the 2.5 % crash rate justifies the bump alone. Only the host knows what else ran on it at 2026-08-30 10:34:49Z and 2026-09-03 07:59:09Z, other repositories' jobs included. What do you give? Options: a) the load at those times, in the answer b) not known: no parallelism cap now; the bump only Recommended: b)

Answer in a comment on this issue; intake reads it on its next iteration.

Intake: needs-input Workflow: diagnostic Reason: G1 (3 CLARIFY markers) — all three need facts from the self-hosted runner host Analysis: project-management/issues/IW-512/analysis.md on branch IW-512 1. [G1] The `format` job crashed with exit 139 in 2 of 79 runs. The fix for Hypothesis 1 (a scalafmt version whose native binary does not crash; `.scalafmt.conf` pins 3.10.4) does not wait for these answers. Hypotheses 2-4 turn on the runner's container setup, which is in the Forgejo runner's `config.yml` on the host, not in this repository: `memory.max` and `cpu.max` of a job container, whether `nproc` inside it reports the host's cores, whether a host directory is mounted as the coursier cache, and the runner's `capacity` (job containers at once). What do you give? Options: a) the four values, in the answer b) the values are not known now: build the Hypothesis 1 fix and the crash instrumentation of Defect 2, which makes the next crash report these facts in the job log, and leave Hypotheses 2-4 open Recommended: b) 2. [G1] Only the host can confirm the cause of the crash of 2026-09-03 07:59:09Z (and 2026-08-30 10:34:49Z): an OOM kill in `dmesg -T` near those times, or the core file of "Segmentation fault (core dumped)" (its place is `kernel.core_pattern`), whose backtrace names the faulting frame. The logs show exit 139 and no OOM message, which already argues against an OOM kill (exit 137) and Scala Native heap exhaustion (exit 1). What do you give? Options: a) what `dmesg` and the core file show, in the answer b) the evidence is gone: rely on the signature match and the Defect 2 instrumentation Recommended: b) 3. [G1] Whether the two crashes correlate with load decides only whether the analysis adds a parallelism cap (Hypothesis 2) beside the version bump; the 2.5 % crash rate justifies the bump alone. Only the host knows what else ran on it at 2026-08-30 10:34:49Z and 2026-09-03 07:59:09Z, other repositories' jobs included. What do you give? Options: a) the load at those times, in the answer b) not known: no parallelism cap now; the bump only Recommended: b) Answer in a comment on this issue; intake reads it on its next iteration.

claude added

needs-input

and removed

needs-human

labels

2026-10-06 18:16:55 +00:00

mph commented

2026-10-10 20:08:31 +00:00

Author

Owner

Drop it now.

mph closed this issue