Integration tests should use a project-controlled CLAUDE_CONFIG_DIR, not the developer's #59

New issue

Open

opened 2026-07-28 10:07:19 +00:00 by mph · 0 comments

mph commented

2026-07-28 10:07:19 +00:00

Owner

Problem

The integration tests spawn a real claude CLI, and almost none of them set CLAUDE_CONFIG_DIR. The spawned process therefore inherits the developer's own config directory (~/.claude, or whatever CLAUDE_CONFIG_DIR happens to be exported in their shell).

That makes the suite non-hermetic in both directions:

Inbound — the test runs under whatever the developer has configured globally: hooks, settings.json, permission mode, MCP servers, model selection, statusline. Two machines can produce different results from the same commit, and a failure may be caused by a setting no reviewer can see from the repo.
Outbound — anything the ambient config wires up fires for real during the test run. Global hooks execute against test sessions, and MCP servers configured by the developer are reachable from them.

Neither depends on what any particular configuration contains. A test suite that spawns a subprocess should decide that subprocess's environment.

Current state

Of 27 files under */itest, exactly one sets the variable — zio/itest/.../ConversationArchiveE2ETest.scala:21-29:

configDir <- ZIO.attempt(os.temp.dir())
options = QueryOptions
            .simple("Reply with exactly one word: pong")
            .withCwd(workDir.toString)
            .withEnvironmentVariables(
              Map("CLAUDE_CONFIG_DIR" -> configDir.toString)
            )

So the pattern and the API (QueryOptions.withEnvironmentVariables / SessionOptions.withEnvironmentVariables, core/src/.../QueryOptions.scala:158) already exist — they are simply not applied consistently. That test needed an isolated config dir to assert on the transcript tree; every other test gets the developer's by default.

Observed instance

While debugging unrelated noise in another project we found ~225 real claude sessions, spawned by this suite from out/<module>/itest/testForked.dest/worker-N/sandbox, had triggered a globally-registered SessionStart/Stop hook belonging to that other project — one per test session, over months of test runs.

Recording it only as evidence that the inbound/outbound coupling is real rather than theoretical. The other project has fixed its side; nothing here is a defect in claude-code-query's behaviour, and this issue would stand even if that hook had never existed.

Suggested direction

A shared itest helper that allocates a temp config dir per test (or per suite) and applies it to every QueryOptions/SessionOptions the tests build, so isolation is the default rather than something each test remembers. Worth considering alongside it:

withInheritEnvironment(false) where a test does not need the ambient environment at all, which would close the same gap for every other inherited variable rather than just this one.
A check that fails loudly if an itest constructs options without a config dir, so the next test added does not silently regress to the developer's.

Naming the temp dirs (os.temp.dir(prefix = ...)) would also make it obvious where a stray sandbox came from when one shows up somewhere unexpected.

## Problem The integration tests spawn a real `claude` CLI, and almost none of them set `CLAUDE_CONFIG_DIR`. The spawned process therefore inherits the developer's own config directory (`~/.claude`, or whatever `CLAUDE_CONFIG_DIR` happens to be exported in their shell). That makes the suite non-hermetic in both directions: - **Inbound** — the test runs under whatever the developer has configured globally: hooks, `settings.json`, permission mode, MCP servers, model selection, statusline. Two machines can produce different results from the same commit, and a failure may be caused by a setting no reviewer can see from the repo. - **Outbound** — anything the ambient config wires up *fires for real* during the test run. Global hooks execute against test sessions, and MCP servers configured by the developer are reachable from them. Neither depends on what any particular configuration contains. A test suite that spawns a subprocess should decide that subprocess's environment. ## Current state Of 27 files under `*/itest`, exactly one sets the variable — `zio/itest/.../ConversationArchiveE2ETest.scala:21-29`: ```scala configDir <- ZIO.attempt(os.temp.dir()) options = QueryOptions .simple("Reply with exactly one word: pong") .withCwd(workDir.toString) .withEnvironmentVariables( Map("CLAUDE_CONFIG_DIR" -> configDir.toString) ) ``` So the pattern and the API (`QueryOptions.withEnvironmentVariables` / `SessionOptions.withEnvironmentVariables`, `core/src/.../QueryOptions.scala:158`) already exist — they are simply not applied consistently. That test needed an isolated config dir to assert on the transcript tree; every other test gets the developer's by default. ## Observed instance While debugging unrelated noise in another project we found ~225 real `claude` sessions, spawned by this suite from `out/<module>/itest/testForked.dest/worker-N/sandbox`, had triggered a globally-registered `SessionStart`/`Stop` hook belonging to that other project — one per test session, over months of test runs. Recording it only as evidence that the inbound/outbound coupling is real rather than theoretical. The other project has fixed its side; nothing here is a defect in `claude-code-query`'s behaviour, and this issue would stand even if that hook had never existed. ## Suggested direction A shared itest helper that allocates a temp config dir per test (or per suite) and applies it to every `QueryOptions`/`SessionOptions` the tests build, so isolation is the default rather than something each test remembers. Worth considering alongside it: - `withInheritEnvironment(false)` where a test does not need the ambient environment at all, which would close the same gap for every other inherited variable rather than just this one. - A check that fails loudly if an itest constructs options without a config dir, so the next test added does not silently regress to the developer's. Naming the temp dirs (`os.temp.dir(prefix = ...)`) would also make it obvious where a stray sandbox came from when one shows up somewhere unexpected.