The dev-mode tests stop the server they start #417

Merged

mph merged 1 commit from fix-orphaned-dev-servers into main

2026-08-13 07:12:31 +00:00

mph commented

2026-08-13 06:38:19 +00:00

Owner

The defect

Each E2E run leaked up to 14 dashboard JVMs. 41 were alive on the development
machine, orphaned onto init, holding their ports and 3.9 GB of RAM. The
oldest had run for 37 hours.

./iw dashboard reaches the server JVM through a chain:

timeout -> iw (exec) -> iw-run (exec) -> scala-cli -> java -jar out.jar ... --dev

iw and iw-run use exec, so a signal reaches scala-cli. scala-cli does
not pass it to the JVM it started, and LiveCommandEnv.startServerAndBlock
starts the server with ProcessBuilder and blocks on waitFor(). The tests
killed the launcher PID, so the server never learned that its parent had died.

The suite-level net did not work either. teardown ran:

pkill -f "dashboard --dev"

No process has that string in its command line. The server's command line is
java -jar out.jar <state> <port> <hosts> --dev. This is the same defect the
IW-345 review recorded for dashboard-dev-gate.bats, which was never corrected.

teardown then ran rm -rf /tmp/iw-dev-* — which deleted the state directory of
the still-running orphan, and would also delete the directory of a developer's
live ./iw dashboard --dev session.

The fix

start_dev_server launches under setsid, so the whole launcher chain gets its
own process group. stop_dev_server signals that group, so the server dies with
its launcher, and removes only the /tmp/iw-dev-<timestamp> directory of the
server it started.

The pair replaces the launch-poll-kill block that was copied into all 12 tests,
together with register_worktree for the registration call they share: 417 lines
deleted, 112 added. The duplication is why one wrong kill was repeated 14 times.

dashboard-dev-gate.bats had the same defect in its teardown — it removed
/tmp/iw-dev-gate-*, which nothing creates, and leaked two directories per run.
Its GATE_SERVER_PID guard was never assigned to.

Verification

The new test stopping the dev server leaves no server process behind starts a
dev server, waits for its health endpoint, stops it, and asserts with pgrep that
no JVM holds its state file. It fails on the old kill path (verified before
the fix, with an orphan left behind) and passes on the new one.

The first version of that test was a false pass: it killed at the "Port:" line,
which the launcher prints before it spawns the server, so there was nothing to
orphan. The health wait is what makes it a real reproduction.

Gate: E2E 177 BATS tests, 0 failures (full run captured and counted).
After the run: 0 orphaned servers, 0 leftover /tmp/iw-dev-* directories —
against up to 14 orphans before. setsid and pgrep were confirmed present in
the CI image.

Not included

The product-side change I first proposed — a JVM shutdown hook in
startServerAndBlock — is not here. It would not have fixed this: the signal
stops at scala-cli and never reaches the JVM that would run the hook. The
paths that matter are already correct — Ctrl+C signals the whole foreground
group, and ./iw server start is a managed daemon with a PID file. Adding the
hook would be code that fixes nothing demonstrable.

🤖 Generated with Claude Code

## The defect Each E2E run leaked up to 14 dashboard JVMs. 41 were alive on the development machine, orphaned onto init, holding their ports and **3.9 GB of RAM**. The oldest had run for 37 hours. `./iw dashboard` reaches the server JVM through a chain: ``` timeout -> iw (exec) -> iw-run (exec) -> scala-cli -> java -jar out.jar ... --dev ``` `iw` and `iw-run` use `exec`, so a signal reaches `scala-cli`. `scala-cli` does not pass it to the JVM it started, and `LiveCommandEnv.startServerAndBlock` starts the server with `ProcessBuilder` and blocks on `waitFor()`. The tests killed the launcher PID, so the server never learned that its parent had died. The suite-level net did not work either. `teardown` ran: ```bash pkill -f "dashboard --dev" ``` No process has that string in its command line. The server's command line is `java -jar out.jar <state> <port> <hosts> --dev`. This is the same defect the IW-345 review recorded for `dashboard-dev-gate.bats`, which was never corrected. `teardown` then ran `rm -rf /tmp/iw-dev-*` — which deleted the state directory of the still-running orphan, and would also delete the directory of a developer's live `./iw dashboard --dev` session. ## The fix `start_dev_server` launches under `setsid`, so the whole launcher chain gets its own process group. `stop_dev_server` signals that group, so the server dies with its launcher, and removes only the `/tmp/iw-dev-<timestamp>` directory of the server it started. The pair replaces the launch-poll-kill block that was copied into all 12 tests, together with `register_worktree` for the registration call they share: 417 lines deleted, 112 added. The duplication is why one wrong kill was repeated 14 times. `dashboard-dev-gate.bats` had the same defect in its teardown — it removed `/tmp/iw-dev-gate-*`, which nothing creates, and leaked two directories per run. Its `GATE_SERVER_PID` guard was never assigned to. ## Verification The new test `stopping the dev server leaves no server process behind` starts a dev server, waits for its health endpoint, stops it, and asserts with `pgrep` that no JVM holds its state file. It **fails on the old kill path** (verified before the fix, with an orphan left behind) and passes on the new one. The first version of that test was a false pass: it killed at the "Port:" line, which the launcher prints *before* it spawns the server, so there was nothing to orphan. The health wait is what makes it a real reproduction. **Gate:** E2E 177 BATS tests, 0 failures (full run captured and counted). After the run: **0 orphaned servers, 0 leftover `/tmp/iw-dev-*` directories** — against up to 14 orphans before. `setsid` and `pgrep` were confirmed present in the CI image. ## Not included The product-side change I first proposed — a JVM shutdown hook in `startServerAndBlock` — is not here. It would not have fixed this: the signal stops at `scala-cli` and never reaches the JVM that would run the hook. The paths that matter are already correct — Ctrl+C signals the whole foreground group, and `./iw server start` is a managed daemon with a PID file. Adding the hook would be code that fixes nothing demonstrable. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mph added 1 commit

2026-08-13 06:38:19 +00:00

test(e2e): stop the dev server by process group so no JVM is orphaned

CI / compile (pull_request) Successful in 1m39s

Details

CI / dashboard-build (pull_request) Successful in 1m49s

Details

CI / test (pull_request) Successful in 14m17s

Details

CI / contract (pull_request) Has been skipped

Details

CI / format (pull_request) Successful in 10s

Details

CI / lint (pull_request) Successful in 11s

Details

74a79c19c7

`./iw dashboard` reaches the server JVM through iw-run and scala-cli, and
neither passes a signal down to it. The tests killed the launcher PID, so
the server survived, was reparented to init, and kept its port and about
100 MB. A full E2E run leaked up to 14 of them; 41 were alive on the
development machine, holding 3.9 GB.

The suite-level net did not work either: `pkill -f "dashboard --dev"`
matches no process, because the server's command line is
`java -jar out.jar <state> <port> <hosts> --dev`.

start_dev_server now launches under `setsid` and stop_dev_server signals
the whole process group, so the server dies with its launcher. Both
replace the launch and kill block that was copied into all 12 tests, with
register_worktree for the registration call they share.

Each test now removes only the /tmp/iw-dev-<timestamp> directory of the
server it started. The wildcard delete it replaces also removed the
directory of a developer's live `./iw dashboard --dev` session, and ran
while the orphan was still reading from it.

dashboard-dev-gate.bats had the same defect in its teardown: it removed
/tmp/iw-dev-gate-*, which nothing creates, and leaked two directories per
run. Its GATE_SERVER_PID guard was never assigned.

Pinned by "stopping the dev server leaves no server process behind",
which fails on the old kill path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

mph merged commit a89b576c6d into main

2026-08-13 07:12:31 +00:00

mph deleted branch fix-orphaned-dev-servers

2026-08-13 07:12:31 +00:00

mph referenced this pull request from a commit

2026-08-13 07:12:32 +00:00

Merge pull request 'The dev-mode tests stop the server they start' (#417) from fix-orphaned-dev-servers into main