ZIO: closing a query or session hangs for ever in a container whose PID 1 does not reap #92

New issue

Open

opened 2026-10-16 13:28:03 +00:00 by claude · 0 comments

claude commented

2026-10-16 13:28:03 +00:00

Member

Problem

In the ZIO API, the end of a query or a session can hang for ever when the process runs in a container whose PID 1 does not reap zombies (docker run without --init, a Jenkins/act-style container that keeps itself alive with tail -f /dev/null or cat, and similar). The hang is in a finalizer, so it is uninterruptible: a Stop does not end it, and the fiber that closes the scope never returns.

Affected: zio only, from v0.6.1 on (the first release with the tree kill). direct and effectful kill the CLI alone. kyo's CliProcess.killTree (kyo/src/works/iterative/claude/kyo/internal/cli/CliProcess.scala:78-90) kills the same tree but does not wait for the descendants, so it does not hang. That is a working example in this repo.
Both call sites: ProcessManager.executeProcess (query) and SessionProcess.start (session) go through ProcessManager.spawnUnderKill (zio/src/works/iterative/claude/zio/internal/cli/ProcessManager.scala:62-69).
The trigger needs a descendant of the CLI (a tool, or the stub's sleep) that dies together with its parent. The parent then never reaps it, the system gives the zombie to PID 1 (or to the nearest subreaper), and that process never calls wait. Whether the parent reaps first is a race, so the failure is intermittent on a host without a reaper, and never shows on a host with systemd/init.

CI is unblocked by #91 (container: init: true). That workaround hides this bug from CI. It does not fix the SDK.

Root cause (proven)

spawnUnderKill registers spawned.killTreeForcibly.ignore as the scope's kill finalizer.

zio-process 0.8.0, zio/process/ProcessPlatformSpecific.scala:76-93:

def killTreeForcibly: ZIO[Any, CommandError, Unit] =
  self.execute { process =>
    val d = process.descendants().collect(Collectors.toList()).asScala
    d.foreach { p => destroyForciblyHandle(p); () }
    destroyForciblyUnsafe
    waitForUnsafe
    d.foreach { p =>
      if (isAliveHandle(p)) {
        onExitHandle(p).get // `ProcessHandle` doesn't have waitFor
        ()
      }
    }
  }

It blocks on onExit().get for every descendant that is still "alive", with no timeout.

For a process that is not its child, the JDK cannot wait. It polls liveness, and a zombie counts as alive: JDK-8284282 "ProcessHandle/OnExitTest times out when run in misconfigured container", Closed / Won't Fix: "the test will wait for them forever, since ProcessHandle.isAlive() returns true for zombified processes". JDK-8284874 only added a comment to that test. The JDK will not change this.
- Not checked by us (Perplexity's reading of the source): ProcessHandleImpl_linux.c os_getParentPidAndTimings reads /proc/<pid>/stat and skips the state field, so state Z is never looked at.
The finalizer runs uninterruptibly, so the wait cannot be cut from outside.

Evidence

CI: SessionCloseTest ("a CLI that never exits is killed once the ceiling passes", sometimes also "a close that is an interruption kills the CLI without waiting") hit its 60 s timeout on the main publish run for 807cc2c (run 1371), on PR #89/#90 (runs 1391, 1392), and on PR #86 commits (e.g. run 1291). It passed on some runs (e.g. 1296): the race.
The runner's job container is docker run … entrypoint=["tail" "-f" "/dev/null"] (runner v13.1.0, image ghcr.io/catthehacker/ubuntu:act-22.04).

Local reproduction, deterministic (0 of 2 tests pass, every run). The wrapper below makes itself a child subreaper and never reaps orphans:

# nonreaping.py — run a command under a subreaper that never reaps orphans
import ctypes, os, sys
PR_SET_CHILD_SUBREAPER = 36
assert ctypes.CDLL(None, use_errno=True).prctl(PR_SET_CHILD_SUBREAPER, 1, 0, 0, 0) == 0
pid = os.fork()
if pid == 0:
    os.execvp(sys.argv[1], sys.argv[1:])
_, status = os.waitpid(pid, 0)  # reaps the direct child only; orphans stay zombies
sys.exit(os.waitstatus_to_exitcode(status))

python3 -I nonreaping.py ./mill --no-daemon zio.test.testOnly '*SessionCloseTest'

--no-daemon is needed: otherwise the test JVM runs under the Mill daemon, outside the wrapper. Without the wrapper the same command passes 3 of 3 on a desktop host.

During the hang: ps shows sleep zombies whose PPID is the wrapper. jstack of the test JVM shows both close threads at CompletableFuture.get ← zio.process.ProcessPlatformSpecific.killTreeForcibly…(ProcessPlatformSpecific.scala:89).

What we checked upstream

zio/zio-process has no issue or PR about this. The only related items are #182, #200 and #209, which added killTree/killTreeForcibly. Open #425 (linesStream is not interruptible) is a different problem.
The kill code has not changed since 32715ed (2025-05-17, #489). v0.8.1 (2026-09-10) exists and does not change it. An upgrade does not fix this.
Possible follow-up: report it upstream, with JDK-8284282 and the reproduction. Not done yet (Michal to decide).

CI facts learned (so nobody probes them again)

Forgejo 16.0.1, runner v13.1.0. Without a container: key the job container's PID 1 is tail -f /dev/null.
container: with only init: true (no image) keeps the image of the runs-on label (runner RunContext.platformImage, act/runner/run_context.go:1379-1395) and starts /sbin/docker-init -- tail -f /dev/null as PID 1. An orphan is then reaped.
init must be a literal. init: ${{ matrix.init }} is dropped silently: the runner decodes the container mapping (Init *bool, act/model/workflow.go:638) before it evaluates expressions, the decode fails, and the default false applies. The job log shows the same docker run line either way. Only PID 1 (ps -p 1) tells the difference.
ci.yml's itest job already solves the same problem in another way: tini -s -- ./mill --no-daemon __.itest (tini as child subreaper).
Probe evidence: branch ci/init-probe (throwaway workflow init-probe.yml), run 1417 (matrix: both still tail) and the literal re-run (docker-init, 2 of 2 pass).

Proposed fix

Replace killTreeForcibly in ProcessManager.spawnUnderKill with our own tree kill:

Collect the descendants while the CLI is still alive, then destroyForcibly them and the CLI. Same order as kyo's killTree.
Wait for the CLI with waitFor. It is our direct child, so the JVM reaps it and this wait always ends.
Wait for each descendant to be finished, up to a short bound (for example 2 s, a named constant). On Linux, a descendant whose state in /proc/<pid>/stat is Z counts as finished: SIGKILL has ended it, and only its process-table entry is left, which only its adopter can remove. Without this check every close in such a container takes the full bound. Elsewhere (macOS: no /proc), use isAlive alone.
Log one warning that names each descendant still alive after the bound, and go on. The kill must never block the scope without a limit.

Points to keep: the kill stays the first finalizer registered (so it runs last), and stays uninterruptible as one step with the spawn (see the scaladoc of spawnUnderKill and the 0.6.1 release commit 0239d1c). The bound must be shorter than SessionCloseTest's took < 5.seconds.

Also check the PID reuse question: between the kill and the check, a PID can in theory be reused. ProcessHandle keeps the start time and compares it. A /proc/<pid>/stat read must compare the start time too (field 22), or keep the ProcessHandle and read /proc only for the state.

Test plan

Unit: put "is this descendant finished?" behind a small function and test the bounded wait with a ProcessHandle that never exits (a JDK interface, not our code). Cases: a descendant that exits in time; one that never exits (returns after the bound, warns); one reported as zombie (returns at once).
Failing test first, in the real environment: a CI step (or job) that runs SessionCloseTest under the non-reaping wrapper above, or a job with no container: block. It must fail on main as it is today, and pass after the fix. Keep it after the fix, because #91 removed the only environment where this bug showed.
In-JVM alternative, not chosen: the test JVM could make itself a subreaper with prctl through the FFM API. FFM is final in Java 22 and preview in Java 21, and CI runs Java 21 (the kyo module already runs on Java 25, so it is possible but is a larger change).

Open questions

Report upstream to zio-process (issue, or a PR with a bounded wait)?
After the fix, keep container: init: true in CI (#91), or remove it and let the no-init job be the default?
Does the 0.6.1 claim "the CLI and the tools it started are gone within ~150 ms" need a note in the README for containers without an init?

## Problem In the ZIO API, the end of a query or a session can hang **for ever** when the process runs in a container whose PID 1 does not reap zombies (`docker run` without `--init`, a Jenkins/act-style container that keeps itself alive with `tail -f /dev/null` or `cat`, and similar). The hang is in a finalizer, so it is uninterruptible: a Stop does not end it, and the fiber that closes the scope never returns. - Affected: `zio` only, from **v0.6.1** on (the first release with the tree kill). `direct` and `effectful` kill the CLI alone. `kyo`'s `CliProcess.killTree` (`kyo/src/works/iterative/claude/kyo/internal/cli/CliProcess.scala:78-90`) kills the same tree but does not wait for the descendants, so it does not hang. That is a working example in this repo. - Both call sites: `ProcessManager.executeProcess` (query) and `SessionProcess.start` (session) go through `ProcessManager.spawnUnderKill` (`zio/src/works/iterative/claude/zio/internal/cli/ProcessManager.scala:62-69`). - The trigger needs a descendant of the CLI (a tool, or the stub's `sleep`) that dies together with its parent. The parent then never reaps it, the system gives the zombie to PID 1 (or to the nearest subreaper), and that process never calls `wait`. Whether the parent reaps first is a race, so the failure is intermittent on a host without a reaper, and never shows on a host with systemd/init. CI is unblocked by #91 (`container: init: true`). That workaround hides this bug from CI. It does not fix the SDK. ## Root cause (proven) 1. `spawnUnderKill` registers `spawned.killTreeForcibly.ignore` as the scope's kill finalizer. 2. zio-process **0.8.0**, `zio/process/ProcessPlatformSpecific.scala:76-93`: ```scala def killTreeForcibly: ZIO[Any, CommandError, Unit] = self.execute { process => val d = process.descendants().collect(Collectors.toList()).asScala d.foreach { p => destroyForciblyHandle(p); () } destroyForciblyUnsafe waitForUnsafe d.foreach { p => if (isAliveHandle(p)) { onExitHandle(p).get // `ProcessHandle` doesn't have waitFor () } } } ``` It blocks on `onExit().get` for every descendant that is still "alive", with no timeout. 3. For a process that is not its child, the JDK cannot `wait`. It polls liveness, and **a zombie counts as alive**: **JDK-8284282** "ProcessHandle/OnExitTest times out when run in misconfigured container", Closed / **Won't Fix**: "the test will wait for them forever, since `ProcessHandle.isAlive()` returns true for zombified processes". JDK-8284874 only added a comment to that test. The JDK will not change this. - Not checked by us (Perplexity's reading of the source): `ProcessHandleImpl_linux.c` `os_getParentPidAndTimings` reads `/proc/<pid>/stat` and skips the state field, so state `Z` is never looked at. 4. The finalizer runs uninterruptibly, so the wait cannot be cut from outside. ### Evidence - CI: `SessionCloseTest` ("a CLI that never exits is killed once the ceiling passes", sometimes also "a close that is an interruption kills the CLI without waiting") hit its 60 s timeout on the main publish run for 807cc2c (run 1371), on PR #89/#90 (runs 1391, 1392), and on PR #86 commits (e.g. run 1291). It passed on some runs (e.g. 1296): the race. - The runner's job container is `docker run … entrypoint=["tail" "-f" "/dev/null"]` (runner v13.1.0, image `ghcr.io/catthehacker/ubuntu:act-22.04`). - Local reproduction, deterministic (0 of 2 tests pass, every run). The wrapper below makes itself a child subreaper and never reaps orphans: ```python # nonreaping.py — run a command under a subreaper that never reaps orphans import ctypes, os, sys PR_SET_CHILD_SUBREAPER = 36 assert ctypes.CDLL(None, use_errno=True).prctl(PR_SET_CHILD_SUBREAPER, 1, 0, 0, 0) == 0 pid = os.fork() if pid == 0: os.execvp(sys.argv[1], sys.argv[1:]) _, status = os.waitpid(pid, 0) # reaps the direct child only; orphans stay zombies sys.exit(os.waitstatus_to_exitcode(status)) ``` ``` python3 -I nonreaping.py ./mill --no-daemon zio.test.testOnly '*SessionCloseTest' ``` `--no-daemon` is needed: otherwise the test JVM runs under the Mill daemon, outside the wrapper. Without the wrapper the same command passes 3 of 3 on a desktop host. - During the hang: `ps` shows `sleep` zombies whose PPID is the wrapper. `jstack` of the test JVM shows both close threads at `CompletableFuture.get` ← `zio.process.ProcessPlatformSpecific.killTreeForcibly…(ProcessPlatformSpecific.scala:89)`. ## What we checked upstream - zio/zio-process has **no** issue or PR about this. The only related items are #182, #200 and #209, which added `killTree`/`killTreeForcibly`. Open #425 (`linesStream` is not interruptible) is a different problem. - The kill code has not changed since 32715ed (2025-05-17, #489). **v0.8.1** (2026-09-10) exists and does not change it. An upgrade does not fix this. - Possible follow-up: report it upstream, with JDK-8284282 and the reproduction. Not done yet (Michal to decide). ## CI facts learned (so nobody probes them again) - Forgejo 16.0.1, runner v13.1.0. Without a `container:` key the job container's PID 1 is `tail -f /dev/null`. - `container:` with only `init: true` (no `image`) keeps the image of the `runs-on` label (runner `RunContext.platformImage`, `act/runner/run_context.go:1379-1395`) and starts `/sbin/docker-init -- tail -f /dev/null` as PID 1. An orphan is then reaped. - **`init` must be a literal.** `init: ${{ matrix.init }}` is dropped silently: the runner decodes the `container` mapping (`Init *bool`, `act/model/workflow.go:638`) before it evaluates expressions, the decode fails, and the default `false` applies. The job log shows the same `docker run` line either way. Only PID 1 (`ps -p 1`) tells the difference. - `ci.yml`'s `itest` job already solves the same problem in another way: `tini -s -- ./mill --no-daemon __.itest` (tini as child subreaper). - Probe evidence: branch `ci/init-probe` (throwaway workflow `init-probe.yml`), run 1417 (matrix: both still `tail`) and the literal re-run (docker-init, 2 of 2 pass). ## Proposed fix Replace `killTreeForcibly` in `ProcessManager.spawnUnderKill` with our own tree kill: 1. Collect the descendants while the CLI is still alive, then `destroyForcibly` them and the CLI. Same order as `kyo`'s `killTree`. 2. Wait for the CLI with `waitFor`. It is our direct child, so the JVM reaps it and this wait always ends. 3. Wait for each descendant to be **finished**, up to a short bound (for example 2 s, a named constant). On Linux, a descendant whose state in `/proc/<pid>/stat` is `Z` counts as finished: SIGKILL has ended it, and only its process-table entry is left, which only its adopter can remove. Without this check every close in such a container takes the full bound. Elsewhere (macOS: no `/proc`), use `isAlive` alone. 4. Log one warning that names each descendant still alive after the bound, and go on. The kill must never block the scope without a limit. Points to keep: the kill stays the first finalizer registered (so it runs last), and stays uninterruptible as one step with the spawn (see the scaladoc of `spawnUnderKill` and the 0.6.1 release commit 0239d1c). The bound must be shorter than `SessionCloseTest`'s `took < 5.seconds`. Also check the PID reuse question: between the kill and the check, a PID can in theory be reused. `ProcessHandle` keeps the start time and compares it. A `/proc/<pid>/stat` read must compare the start time too (field 22), or keep the `ProcessHandle` and read `/proc` only for the state. ## Test plan - **Unit:** put "is this descendant finished?" behind a small function and test the bounded wait with a `ProcessHandle` that never exits (a JDK interface, not our code). Cases: a descendant that exits in time; one that never exits (returns after the bound, warns); one reported as zombie (returns at once). - **Failing test first, in the real environment:** a CI step (or job) that runs `SessionCloseTest` under the non-reaping wrapper above, or a job with no `container:` block. It must fail on main as it is today, and pass after the fix. Keep it after the fix, because #91 removed the only environment where this bug showed. - In-JVM alternative, not chosen: the test JVM could make itself a subreaper with `prctl` through the FFM API. FFM is final in Java 22 and preview in Java 21, and CI runs Java 21 (the `kyo` module already runs on Java 25, so it is possible but is a larger change). ## Open questions - Report upstream to zio-process (issue, or a PR with a bounded wait)? - After the fix, keep `container: init: true` in CI (#91), or remove it and let the no-init job be the default? - Does the 0.6.1 claim "the CLI and the tools it started are gone within ~150 ms" need a note in the README for containers without an init?

claude referenced this issue from a commit

2026-10-16 13:41:20 +00:00

Merge pull request 'ci: give the jobs that run the unit suite a reaping PID 1' (#91) from ci/reaping-init into main

claude referenced this issue

2026-10-16 13:42:42 +00:00

Release 0.7.0 #90