ProcessAdapter.run cannot spawn during JVM shutdown, so no recovery step can reach git #616

New issue

Closed

opened 2026-09-24 19:11:17 +00:00 by claude · 0 comments

claude commented

2026-09-24 19:11:17 +00:00

Member

What happens

ProcessAdapter.run (core/adapters/Process.scala:66-83) reaches every external command through os.proc(...).call(...). os-lib registers a JVM shutdown hook for each spawned child:

// os-lib 0.11.6, ProcessOps.scala:326-342
lazy val shutdownHookThread =
  if (!destroyOnExit) None
  else Some(new Thread("subprocess-shutdown-hook") {
    override def run(): Unit = proc.destroy(shutdownGracePeriod)
  })
...
shutdownHookThread.foreach(Runtime.getRuntime().addShutdownHook)

Runtime.addShutdownHook throws IllegalStateException: Shutdown in progress once shutdown has begun. So no caller of ProcessAdapter.run can run a command while the JVM is shutting down — the call does not fail, it throws, and the throw escapes any Either the caller wrapped it in.

Why it matters

A recovery step is exactly the code that runs while the process is dying, and it is the code that most needs to reach git.

kanon's batch-implement hit this twice on 2026-09-23. The host killed a phase session at exit 137, the signal reached the whole process group, and the driver entered recovery with shutdown already begun:

[phase 2] claude exited with code 137 (entering recovery)
Exception in thread "main" java.lang.IllegalStateException: Shutdown in progress
  at java.lang.Runtime.addShutdownHook(Runtime.java:250)
  at os.proc.spawn(ProcessOps.scala:342)
  at iw.core.adapters.ProcessAdapter$.run(Process.scala:76)
  at batch.PhaseRecovery$.commitLeftovers(PhaseRecovery.scala:81)

The second kill left about 600 lines of finished work uncommitted. commitLeftovers exists to save exactly that, and it could not run. The work was committed by hand.

The condition the recovery step is written for is the condition in which it cannot spawn.

Requested

Give ProcessAdapter.run a destroyOnExit: Boolean = true parameter and pass it to os.proc.call. The default keeps today's behaviour for every existing caller. A caller that must survive shutdown — a recovery or cleanup step — passes false, os-lib registers no hook, and the command runs.

destroyOnExit = false also means the child is not killed when the JVM exits, which is the wanted behaviour here: a git commit that outlives the dying driver by a moment is the point.

Notes

This is the interface change alone. The kanon side is dev-docs #346, which passes false from commitLeftovers and also catches the throw, so a spawn that still fails becomes a warning naming the uncommitted files rather than a stack trace.
A test can prove it without a real kill: Runtime.getRuntime().addShutdownHook throwing is observable by calling run from inside a shutdown hook. A run with destroyOnExit = false completes there; a run with the default throws.

Provenance

Found while implementing dev-docs #346. The two directions that issue proposed both fail: registering our own shutdown hook earlier does not stop os-lib from registering its own per spawn.

## What happens `ProcessAdapter.run` (`core/adapters/Process.scala:66-83`) reaches every external command through `os.proc(...).call(...)`. os-lib registers a JVM shutdown hook for each spawned child: // os-lib 0.11.6, ProcessOps.scala:326-342 lazy val shutdownHookThread = if (!destroyOnExit) None else Some(new Thread("subprocess-shutdown-hook") { override def run(): Unit = proc.destroy(shutdownGracePeriod) }) ... shutdownHookThread.foreach(Runtime.getRuntime().addShutdownHook) `Runtime.addShutdownHook` throws `IllegalStateException: Shutdown in progress` once shutdown has begun. So **no caller of `ProcessAdapter.run` can run a command while the JVM is shutting down** — the call does not fail, it throws, and the throw escapes any `Either` the caller wrapped it in. ## Why it matters A recovery step is exactly the code that runs while the process is dying, and it is the code that most needs to reach git. kanon's `batch-implement` hit this twice on 2026-09-23. The host killed a phase session at exit 137, the signal reached the whole process group, and the driver entered recovery with shutdown already begun: [phase 2] claude exited with code 137 (entering recovery) Exception in thread "main" java.lang.IllegalStateException: Shutdown in progress at java.lang.Runtime.addShutdownHook(Runtime.java:250) at os.proc.spawn(ProcessOps.scala:342) at iw.core.adapters.ProcessAdapter$.run(Process.scala:76) at batch.PhaseRecovery$.commitLeftovers(PhaseRecovery.scala:81) The second kill left about 600 lines of finished work uncommitted. `commitLeftovers` exists to save exactly that, and it could not run. The work was committed by hand. The condition the recovery step is written for is the condition in which it cannot spawn. ## Requested Give `ProcessAdapter.run` a `destroyOnExit: Boolean = true` parameter and pass it to `os.proc.call`. The default keeps today's behaviour for every existing caller. A caller that must survive shutdown — a recovery or cleanup step — passes `false`, os-lib registers no hook, and the command runs. `destroyOnExit = false` also means the child is not killed when the JVM exits, which is the wanted behaviour here: a `git commit` that outlives the dying driver by a moment is the point. ## Notes - This is the interface change alone. The kanon side is dev-docs #346, which passes `false` from `commitLeftovers` and also catches the throw, so a spawn that still fails becomes a warning naming the uncommitted files rather than a stack trace. - A test can prove it without a real kill: `Runtime.getRuntime().addShutdownHook` throwing is observable by calling `run` from inside a shutdown hook. A run with `destroyOnExit = false` completes there; a run with the default throws. ## Provenance Found while implementing dev-docs #346. The two directions that issue proposed both fail: registering our own shutdown hook earlier does not stop os-lib from registering its own per spawn.

claude referenced this issue from a pull request that will close it,

2026-09-24 19:57:48 +00:00

IW-616: Shutdown-safe ProcessAdapter.run #618