PekkoProjectionSupport: exactlyOnce does not cover the read-model write — projections are at-least-once against their own read model #41

New issue

Open

opened 2026-07-31 07:03:56 +00:00 by mph · 0 comments

mph commented

2026-07-31 07:03:56 +00:00

Owner

Summary

PekkoProjectionSupport builds every projection with SlickProjection.exactlyOnce, but the
ViewProcessor write is escaped out of the Slick transaction. The exactly-once guarantee therefore
covers only the offset row — not the read model the projection exists to maintain. Any projection
whose ViewProcessor writes to a store other than the projection's own Slick DatabaseConfig
(MongoDB, HTTP, a second datasource) is at-least-once with respect to that store.

This is not a theoretical concern. It caused a production incident in a downstream project
(medeca-modul-poptavky, MEDECA-411): a re-delivered envelope re-applied a non-idempotent event to an
already-updated read model, the aggregate's event-application threw, and orDieWith turned it into a
ZIO defect that pinned a ShardedDaemonProcess projection at one offset while sibling projections ran
thousands of events ahead. The read model froze and could never catch up.

Where

pekko-persistence/src/main/scala/works/iterative/pekko/PekkoProjectionSupport.scala

proj <- ZIO.attempt {
    SlickProjection.exactlyOnce(
        projectionId = ProjectionId(projectionName, projectionKey),
        sourceProvider,
        dbConfig,
        handler = () => new ProjectionHandler(dbConfig, processor, transform)
    )
}

and, in ProjectionHandler:

override def process(envelope: EventEnvelope[J]): DBIO[Done] =
    DBIO.from(
        Unsafe.unsafe(implicit unsafe =>
            runtime.unsafe.runToFuture(processor.process(transform(envelope.event)).as(Done))
        )
    )

DBIO.from(future) wraps an already-running Future. The effect starts when
runtime.unsafe.runToFuture is called and completes on its own; the surrounding Slick transaction
neither sequences it nor rolls it back. exactlyOnce commits the read-model write and the offset row
atomically only when both are DBIO actions against the same dbConfig. Here the processor's write
is arbitrary ZIO against an arbitrary store.

Consequence: a crash, redeploy, node restart or shard rebalance between the processor's write and the
offset commit re-delivers the envelope, and the processor sees it twice.

Why it is easy to get wrong

The API reads as a guarantee. A caller passing a ViewProcessor that writes to Mongo gets a
projection named exactlyOnce that is not exactly-once for their data, with nothing in the types or
the docs to say so. Every downstream ViewProcessor must be idempotent, and today nothing states that
requirement or checks it.

Compounding factors observed downstream:

numberOfInstances = 1 under ShardedDaemonProcess means one projection instance serves every
entity, so one poisoned envelope stops the read model for all of them, not just the offending
aggregate.
Under ProjectionBehavior supervision with the default recovery strategy, the failing envelope is
retried forever. The projection never advances and never gives up.

Options

Document the contract (cheapest, and worth doing regardless). State in the scaladoc that
ViewProcessor.process must be idempotent whenever it writes outside dbConfig, and that
exactlyOnce guarantees only offset/Slick atomicity. Name the failure mode.
Rename or split the API so the guarantee is honest — e.g. expose atLeastOnce for
foreign-store processors and reserve exactlyOnce for processors that return a DBIO against the
same dbConfig.
Let the processor participate in the transaction — change ViewProcessor so a processor
writing to the projection's own database can return a DBIO that composes into the Slick
transaction instead of being escaped through DBIO.from.
Surface the recovery strategy and instance count as explicit configuration at the call site,
so a caller chooses "stop the world on a poisoned envelope" deliberately rather than inheriting it.

Option 1 is the immediate one. Options 2–4 are a design discussion.

Notes

Applies equally to the Akka variant (AkkaProjectionSupport) — the same DBIO.from escape is
present there. This is not a Pekko-migration regression.
Downstream mitigation in MEDECA-411 was to make the consuming aggregate's event application total
and idempotent. That is the correct fix on the consumer side, but it is a fix each consumer has to
discover independently — currently by having a production projection wedge.

## Summary `PekkoProjectionSupport` builds every projection with `SlickProjection.exactlyOnce`, but the `ViewProcessor` write is escaped out of the Slick transaction. The exactly-once guarantee therefore covers only the offset row — **not** the read model the projection exists to maintain. Any projection whose `ViewProcessor` writes to a store other than the projection's own Slick `DatabaseConfig` (MongoDB, HTTP, a second datasource) is **at-least-once** with respect to that store. This is not a theoretical concern. It caused a production incident in a downstream project (medeca-modul-poptavky, MEDECA-411): a re-delivered envelope re-applied a non-idempotent event to an already-updated read model, the aggregate's event-application threw, and `orDieWith` turned it into a ZIO defect that pinned a `ShardedDaemonProcess` projection at one offset while sibling projections ran thousands of events ahead. The read model froze and could never catch up. ## Where `pekko-persistence/src/main/scala/works/iterative/pekko/PekkoProjectionSupport.scala` ```scala proj <- ZIO.attempt { SlickProjection.exactlyOnce( projectionId = ProjectionId(projectionName, projectionKey), sourceProvider, dbConfig, handler = () => new ProjectionHandler(dbConfig, processor, transform) ) } ``` and, in `ProjectionHandler`: ```scala override def process(envelope: EventEnvelope[J]): DBIO[Done] = DBIO.from( Unsafe.unsafe(implicit unsafe => runtime.unsafe.runToFuture(processor.process(transform(envelope.event)).as(Done)) ) ) ``` `DBIO.from(future)` wraps an **already-running** Future. The effect starts when `runtime.unsafe.runToFuture` is called and completes on its own; the surrounding Slick transaction neither sequences it nor rolls it back. `exactlyOnce` commits the read-model write and the offset row atomically *only* when both are DBIO actions against the same `dbConfig`. Here the processor's write is arbitrary ZIO against an arbitrary store. Consequence: a crash, redeploy, node restart or shard rebalance between the processor's write and the offset commit re-delivers the envelope, and the processor sees it twice. ## Why it is easy to get wrong The API reads as a guarantee. A caller passing a `ViewProcessor` that writes to Mongo gets a projection named `exactlyOnce` that is not exactly-once for their data, with nothing in the types or the docs to say so. Every downstream `ViewProcessor` must be idempotent, and today nothing states that requirement or checks it. Compounding factors observed downstream: - `numberOfInstances = 1` under `ShardedDaemonProcess` means one projection instance serves every entity, so one poisoned envelope stops the read model for **all** of them, not just the offending aggregate. - Under `ProjectionBehavior` supervision with the default recovery strategy, the failing envelope is retried forever. The projection never advances and never gives up. ## Options 1. **Document the contract** (cheapest, and worth doing regardless). State in the scaladoc that `ViewProcessor.process` must be idempotent whenever it writes outside `dbConfig`, and that `exactlyOnce` guarantees only offset/Slick atomicity. Name the failure mode. 2. **Rename or split the API** so the guarantee is honest — e.g. expose `atLeastOnce` for foreign-store processors and reserve `exactlyOnce` for processors that return a `DBIO` against the same `dbConfig`. 3. **Let the processor participate in the transaction** — change `ViewProcessor` so a processor writing to the projection's own database can return a `DBIO` that composes into the Slick transaction instead of being escaped through `DBIO.from`. 4. **Surface the recovery strategy and instance count** as explicit configuration at the call site, so a caller chooses "stop the world on a poisoned envelope" deliberately rather than inheriting it. Option 1 is the immediate one. Options 2–4 are a design discussion. ## Notes - Applies equally to the Akka variant (`AkkaProjectionSupport`) — the same `DBIO.from` escape is present there. This is not a Pekko-migration regression. - Downstream mitigation in MEDECA-411 was to make the consuming aggregate's event application total and idempotent. That is the correct fix on the consumer side, but it is a fix each consumer has to discover independently — currently by having a production projection wedge.