Add graceful cleanup of build tool daemons when removing worktrees #225

New issue

Closed

opened 2026-02-25 14:08:43 +00:00 by mprihoda · 1 comment

mprihoda commented

2026-02-25 14:08:43 +00:00

(Migrated from github.com)

Problem

When worktrees are removed (via iw worktree remove or manual cleanup), build tool daemon processes spawned inside those worktrees are left running. These orphaned processes accumulate over time, consuming significant CPU and memory.

Observed today: 14 orphaned Mill daemon and Bloop server processes from deleted/stale worktrees were consuming ~2775% CPU (on a 24-core machine) and ~50% RAM combined. The processes had been accumulating since Feb 18 without any cleanup.

Affected build tools (at minimum):

Mill — spawns persistent daemon processes in <project>/out/mill-daemon/sandbox
Bloop — scala-cli's build server, runs at ~/.local/share/scalacli/bloop
sbt — may also leave BSP/Bloop processes (less commonly observed)

Proposed Solution

Add a cleanup step to the worktree removal workflow that:

Detects running build daemons associated with the worktree being removed (by matching CWD or process arguments against the worktree path)
Gracefully terminates them (SIGTERM first, SIGKILL after timeout)
Cleans up daemon state directories (e.g., out/mill-daemon/ for Mill)

Possible approaches

Run mill --no-server shutdown (or equivalent) before removing the worktree directory
Use bloop exit to stop Bloop servers tied to the worktree
As a fallback, find and kill processes whose CWD or cmdline references the worktree path

Edge cases to consider

Worktree directory already deleted (orphaned processes with (deleted) CWD) — need to match by path substring in /proc/<pid>/cmdline
Shared Bloop instances serving multiple projects
Processes that ignore SIGTERM (need SIGKILL fallback with timeout)

Context

This came up while working on the medeca-modul-poptavky project (sbt-based), where stale Mill/Bloop processes from the unmz-evaluation-ai project (Mill-based) were hogging the machine.

## Problem When worktrees are removed (via `iw worktree remove` or manual cleanup), build tool daemon processes spawned inside those worktrees are left running. These orphaned processes accumulate over time, consuming significant CPU and memory. Observed today: 14 orphaned Mill daemon and Bloop server processes from deleted/stale worktrees were consuming ~2775% CPU (on a 24-core machine) and ~50% RAM combined. The processes had been accumulating since Feb 18 without any cleanup. Affected build tools (at minimum): - **Mill** — spawns persistent daemon processes in `<project>/out/mill-daemon/sandbox` - **Bloop** — scala-cli's build server, runs at `~/.local/share/scalacli/bloop` - **sbt** — may also leave BSP/Bloop processes (less commonly observed) ## Proposed Solution Add a cleanup step to the worktree removal workflow that: 1. **Detects running build daemons** associated with the worktree being removed (by matching CWD or process arguments against the worktree path) 2. **Gracefully terminates them** (SIGTERM first, SIGKILL after timeout) 3. **Cleans up daemon state directories** (e.g., `out/mill-daemon/` for Mill) ### Possible approaches - Run `mill --no-server shutdown` (or equivalent) before removing the worktree directory - Use `bloop exit` to stop Bloop servers tied to the worktree - As a fallback, find and kill processes whose CWD or cmdline references the worktree path ### Edge cases to consider - Worktree directory already deleted (orphaned processes with `(deleted)` CWD) — need to match by path substring in `/proc/<pid>/cmdline` - Shared Bloop instances serving multiple projects - Processes that ignore SIGTERM (need SIGKILL fallback with timeout) ## Context This came up while working on the `medeca-modul-poptavky` project (sbt-based), where stale Mill/Bloop processes from the `unmz-evaluation-ai` project (Mill-based) were hogging the machine.

mprihoda commented

2026-06-12 16:23:34 +00:00

(Migrated from github.com)

Superseded by #382, which consolidates this (build-tool-daemon cleanup) with #147 (project-defined hook surface) into a single CleanupAction design. The Mill/Bloop/docker-compose detection proposed here lives in #382 as a built-in BuildToolCleanup default hook so the "hogging CPU after rm" case is fixed with no per-project config. Implementation will land on #382.

Superseded by #382, which consolidates this (build-tool-daemon cleanup) with #147 (project-defined hook surface) into a single `CleanupAction` design. The Mill/Bloop/docker-compose detection proposed here lives in #382 as a built-in `BuildToolCleanup` default hook so the "hogging CPU after rm" case is fixed with no per-project config. Implementation will land on #382.