build.mill instruments Scoverage unconditionally, and Scala 3.3.8 would make that grow without bound #565

New issue

Open

opened 2026-09-15 09:28:13 +00:00 by claude · 1 comment

claude commented

2026-09-15 09:28:13 +00:00

Member

build.mill instruments for Scoverage unconditionally. Three modules extend ScoverageTests, so
every local ./iw ./test writes coverage data:

out/core/scoverage/data.dest/scoverage.coverage        0.2 MB
out/dashboard/scoverage/data.dest/scoverage.coverage   0.6 MB

Those files are small and stable today for one reason only: we are on Scala 3.3.7.

The defect this exposes us to

A Scala 3 coverage defect doubles the backslashes in scoverage.coverage on every incremental
compile:

InstrumentCoverage.runOn re-reads the previous scoverage.coverage and adds back the
statements of every source file not in the current run (scala/scala3#24773).
Serializer.serialize writes those fields through escaped, which turns \ into \\.
Serializer.deserialize reads them as they are and never reverses escaped.

Each compile therefore adds one escaping level to every carried-forward statement. The growth is
exponential in the number of incremental compiles that do not recompile a file.

3.3.7 does not reproduce it. The carry-forward arrived in 3.8.2, and on the LTS line in
3.3.8. So a routine patch bump arms this.

The full analysis, with a compiler-only reproducer, is in the procedures repository at
docs/scala3-coverage-backslash-growth.md.

What it cost in `procedures`

scoverage.coverage reached 48 GB once, and 1.3–6.2 GB per module across six of eight worktrees,
17.7 GiB in total, with single statement descriptions of 55 MB carrying runs of 2^21 backslashes.
The build dies as OutOfMemoryError: Java heap space in Serializer$.deserialize or
Serializer$.escaped — naming neither coverage nor the file — and it dies that way with a 12 GB
heap. Deleting the file repairs the build until it grows again.

iw-cli has the same eight-worktree multiplier.

The fix

Port the gate procedures already uses (commit dfa514de, "build: collect Scoverage coverage only
in CI"): a system property that CI passes and local runs leave unset, so scoverage.compile yields
the module's plain compile result and nothing writes coverage data.

val CoverageEnabled: Boolean =
  BuildCtx.watchValue(sys.props.get("iwcli.coverage").contains("true"))

trait CoverageModule extends ScoverageModule:
  def scoverageVersion = ScoverageVersion
  override lazy val scoverage: ScoverageData = new ScoverageData {
    private def selectedCompile: Task[mill.javalib.api.CompilationResult] =
      if (CoverageEnabled) super.compile else CoverageModule.this.compile
    override def compile: T[mill.javalib.api.CompilationResult] = Task { selectedCompile() }
  }

It is worth doing before the next Scala bump rather than after, because the failure names nothing
that points at coverage. It also removes instrumentation from every local test run, which buys us
nothing today.

Acceptance

A plain ./mill __.test writes no scoverage.coverage.
CI passes the property and the aggregate report is unchanged.
A comment in build.mill names the defect and the version range, so the next person to bump
Scala meets it.

`build.mill` instruments for Scoverage unconditionally. Three modules extend `ScoverageTests`, so every local `./iw ./test` writes coverage data: ``` out/core/scoverage/data.dest/scoverage.coverage 0.2 MB out/dashboard/scoverage/data.dest/scoverage.coverage 0.6 MB ``` Those files are small and stable today for one reason only: we are on **Scala 3.3.7**. ## The defect this exposes us to A Scala 3 coverage defect doubles the backslashes in `scoverage.coverage` on every incremental compile: - `InstrumentCoverage.runOn` re-reads the previous `scoverage.coverage` and adds back the statements of every source file not in the current run (scala/scala3#24773). - `Serializer.serialize` writes those fields through `escaped`, which turns `\` into `\\`. - `Serializer.deserialize` reads them as they are and never reverses `escaped`. Each compile therefore adds one escaping level to every carried-forward statement. The growth is exponential in the number of incremental compiles that do not recompile a file. **3.3.7 does not reproduce it.** The carry-forward arrived in 3.8.2, and on the LTS line in **3.3.8**. So a routine patch bump arms this. The full analysis, with a compiler-only reproducer, is in the `procedures` repository at `docs/scala3-coverage-backslash-growth.md`. ## What it cost in `procedures` `scoverage.coverage` reached 48 GB once, and 1.3–6.2 GB per module across six of eight worktrees, 17.7 GiB in total, with single statement descriptions of 55 MB carrying runs of 2^21 backslashes. The build dies as `OutOfMemoryError: Java heap space` in `Serializer$.deserialize` or `Serializer$.escaped` — naming neither coverage nor the file — and it dies that way with a 12 GB heap. Deleting the file repairs the build until it grows again. iw-cli has the same eight-worktree multiplier. ## The fix Port the gate `procedures` already uses (commit `dfa514de`, "build: collect Scoverage coverage only in CI"): a system property that CI passes and local runs leave unset, so `scoverage.compile` yields the module's plain compile result and nothing writes coverage data. ```scala val CoverageEnabled: Boolean = BuildCtx.watchValue(sys.props.get("iwcli.coverage").contains("true")) trait CoverageModule extends ScoverageModule: def scoverageVersion = ScoverageVersion override lazy val scoverage: ScoverageData = new ScoverageData { private def selectedCompile: Task[mill.javalib.api.CompilationResult] = if (CoverageEnabled) super.compile else CoverageModule.this.compile override def compile: T[mill.javalib.api.CompilationResult] = Task { selectedCompile() } } ``` It is worth doing before the next Scala bump rather than after, because the failure names nothing that points at coverage. It also removes instrumentation from every local test run, which buys us nothing today. ## Acceptance - A plain `./mill __.test` writes no `scoverage.coverage`. - CI passes the property and the aggregate report is unchanged. - A comment in `build.mill` names the defect and the version range, so the next person to bump Scala meets it.

claude commented

2026-10-15 21:34:19 +00:00

Author

Member

Intake: needs-human
Workflow: waterfall
Reason: G4 the analysis proposes a change to CLAUDE.md (the "Coverage (scoverage)" snippet, Documentation Requirements, analysis.md:325)
Analysis: project-management/issues/IW-565/analysis.md on branch IW-565

Intake: needs-human Workflow: waterfall Reason: G4 the analysis proposes a change to CLAUDE.md (the "Coverage (scoverage)" snippet, Documentation Requirements, analysis.md:325) Analysis: project-management/issues/IW-565/analysis.md on branch IW-565

claude added the

needs-human

label

2026-10-15 21:34:19 +00:00