Effects in production

This is the page to read before deploying an effect. Everything here is about what happens after the code is right.

Lanes

One reader thread per effect owns its subscription; the invocations run on a worker pool shared across every effect, bounded by [effects] pool_size (default 16).

An arm’s @key names its lane. One lane processes in log order, and different lanes never wait on each other.

A failing invocation records itself, parks its lane until its backoff expires, and returns its worker. So a wedged lane costs a map entry rather than a thread, and even pool_size = 1 gets the isolation. One unprocessable event stalls one key, not the effect.

position is a low-water mark

This is the single most misread number in the runtime.

position is the highest position every lane has passed. A wedge at position 3 reads as position: 2, lag: 3 even when positions 4 and 5 in other lanes finished long ago.

So lag on a partitioned effect does not mean the effect is behind. It usually means one lane is stuck and everything else is fine.

The five states

Derived in this precedence order:

Quarantine outranks the rest because one restored from an earlier process has a zero failure count. blocked outranks wedged because nothing is retrying. A wedge outranks lag because a wedged effect lags precisely because it is wedged.

Two retry loops

They are deliberately separate, and knowing which is which saves an afternoon.

heklang re-sends on a transport error and on every status that clears on its own: 408, 425, 429 and any 5xx. Those never reach your arm, which is why status >= 400 inside an arm is always a real decision.

hekla retries the invocation when that loop is exhausted or a host call fails: capped exponential backoff from 200 ms doubling to 60 s, forever, replaying journaled calls each attempt so nothing already performed fires again.

A Retry-After on a retryable response raises the invocation’s backoff, capped at five minutes, and never lowers it. The header never reaches a program.

One HTTP attempt is capped at 10 s to connect and 30 s overall. Neither is configurable.

The journal

Every impure call looks itself up first. A recorded result comes back; otherwise the call is performed and its result appended.

Rows commit call by call in autocommit, never one transaction per invocation. That is what makes a crash mid-arm recoverable.

The key is the call itself, for HTTP the verb, URL and body, plus an ordinal separating identical repeated calls, stored as a sha256. It is not a sequence number, so editing or reordering an arm does not corrupt replay: the failure mode of editing during a deploy is “a different path was taken”, not “a payment fired twice”. A duplicate needs an edit to the URL or body of a call that already fired.

log(...) and reveal(...) are not journaled. A duplicated log line is harmless; a replayed reveal would serve stale plaintext against a destroyed key.

Arguments are hashed and never stored, so a journal cannot outlive the erasure it belonged to.

Retention, and why a wedge is urgent

A sweeper runs hourly and deletes completed invocations older than [retention] effect_journal_days (default 7), bounded by the effect’s persisted watermark. Positions above it are exactly what the next boot would replay, and reclaiming them would let their side effects fire twice.

An effect that has never persisted a watermark is never swept at all.

Getting past a wedge

1. Look. pinning_key names the lane, pinning_position is where. Read last_error, then the journaled calls at GET /admin/effects/{Name}/invocations/{pinning_position}. The first missing call is where it is stuck, which is why that list pages.
2. Fix the cause and restart. The running invocation replays, and every completed call comes back from the journal instead of firing again.
3. Skip it, if the event is genuinely unprocessable:

   curl -X POST localhost:8080/effects/NotifyCustomer/skip/412

   Honoured only for a position that has already failed. Several can be pending at once, one per wedged lane, and a request the watermark passes is forgotten. Nothing is ever skipped automatically.
4. Erase the subject a reveal needs. That turns the failure terminal, which also unwedges it.

Terminal failures

Two things complete a position instead of retrying it: the author’s own fail(...), and a reveal of an erased subject. Both increment terminal_skips and set last_terminal_error.

That is the right behaviour for both. Neither can succeed on a later attempt.

blocked

An arm’s @key changed while lanes were still outstanding, so the per-lane rows above the watermark are keyed under a scheme the new key never produces. last_error names the event types whose lane moved and how many lanes are outstanding.

Two ways out, and they are not equivalent:

Redeploy the previous key and let it drain. Watch until wedged_lanes and lag are both 0, then deploy the new key. This costs nothing.

hekla rewind <Effect> <watermark> against a stopped process. It discards the lane rows and the recorded invocations above the watermark, so those positions run again and perform their side effects again. An escape hatch, not the alternative.

hekla plan reports the repartition before the deploy, so this is avoidable rather than something to discover at boot.

on latest

An on latest arm runs once per key per batch, at the newest matching position in it.

Catching up, a key’s whole backlog is one invocation. It is not “skip history”: that one invocation folded the whole prefix, so it acted on the final state. latest_collapsed counts what it folded, which is why such an effect’s lag can fall by far more than its invocation count rises.

rewind

hekla rewind NotifyCustomer 0 .

Moves an effect back to a position so it reprocesses everything after it and performs those side effects again. It refuses while a server holds the data directory, prints what it would discard, and asks before doing it. --yes answers the prompt without silencing the summary; without a TTY and without --yes it refuses.

--live also lowers an on live boundary, and is permanent. It is off by default because the author of an on live arm declared that history is not news, so overriding that should be something you typed.

Rewinding to the current watermark is allowed and is not a no-op: that is the escape hatch a blocked effect needs.

There is deliberately no HTTP equivalent. An on live effect is by definition one where an accidental rewind re-sends every notification the log has ever seen.

Deploys

Graceful shutdown stops dispatching and waits up to 30 seconds for in-flight invocations, so the common deploy has nothing in flight. One abandoned by the timeout stays running and replays at the next start. An in-flight invocation whose recorded source hash differs from the code on disk is logged by name at startup.