Effects in production

This is the page to read before deploying an effect. Everything here is about what happens after the code is right.

Lanes

One reader thread per effect owns its subscription; the invocations run on a worker pool shared across every effect, bounded by [effects] pool_size (default 16).

An arm’s @key names its lane. One lane processes in log order, and different lanes never wait on each other.

A lane belongs to one effect. A lane is the key alone, so two arms of one effect resolving the same key share one, and two effects never do whatever keys they declare: each reads the log on its own cursor, so one key tuple in two effects is two lanes whose positions race. Writers that have to stay in log order against one remote resource therefore belong in one effect, as two arms. An effect is also the unit of quarantine, so that grouping is an operational choice as much as an ordering one.

A failing invocation records itself, parks its lane until its backoff expires, and returns its worker. So a wedged lane costs a map entry rather than a thread, and even pool_size = 1 gets the isolation. One unprocessable event stalls one key, not the effect.

position is a low-water mark

This is the single most misread number in the runtime.

position is the highest position every lane has passed. A wedge at position 3 reads as position: 2, lag: 3 even when positions 4 and 5 in other lanes finished long ago.

So lag on a partitioned effect does not mean the effect is behind. It usually means one lane is stuck and everything else is fine.

stuck_lanes is on GET /admin/effects/{name} and never on the listing, because a listing polled every few seconds across every effect would be mostly error text. It is the only surface a wedge appears on at all: a fail appends an event, so anything reading the log can list one, and a wedge appends nothing.

"stuck_lanes": [
  { "lane": "i:4471", "position": 2, "attempt": 8,
    "error": "effects/send-welcome.hk:11:20: ...", "retry_in_ms": 39719 },
  { "lane": "i:5120", "position": 9, "attempt": 3,
    "error": "effects/send-welcome.hk:11:20: ...", "retry_in_ms": 1200 }
]

Two edges. The array lists at most a hundred lanes, so a wedged_lanes larger than it is the rest being left out, and the two are read together so a gap is never a lane that merely cleared. And pinning_key can be null beside a populated array, which is a driver-level failure outranking every lane, because that one belongs to no partition key.

The five states

Derived in this precedence order:

Quarantine outranks the rest because one restored from an earlier process has a zero failure count. blocked outranks wedged because nothing is retrying. A wedge outranks lag because a wedged effect lags precisely because it is wedged.

Two retry loops

They are deliberately separate, and knowing which is which saves an afternoon.

heklang re-sends on a transport error and on every status that clears on its own: 408, 425, 429 and any 5xx. Those never reach your arm, which is why status >= 400 inside an arm is always a real decision.

hekla retries the invocation when that loop is exhausted or a host call fails: capped exponential backoff from 200 ms doubling to 60 s, forever, replaying journaled calls each attempt so nothing already performed fires again.

A Retry-After on a retryable response raises the invocation’s backoff, capped at five minutes, and never lowers it. The header never reaches a program.

One HTTP attempt is capped at 10 s to connect and 30 s overall. Neither is configurable.

The journal

Every impure call looks itself up first. A recorded result comes back; otherwise the call is performed and its result appended.

Rows commit call by call in autocommit, never one transaction per invocation. That is what makes a crash mid-arm recoverable.

The key is the call itself, for HTTP the verb, URL and body, plus an ordinal separating identical repeated calls, stored as a sha256. It is not a sequence number, so editing or reordering an arm does not corrupt replay: the failure mode of editing during a deploy is “a different path was taken”, not “a payment fired twice”. A duplicate needs an edit to the URL or body of a call that already fired.

log(...) and reveal(...) are not journaled. A duplicated log line is harmless; a replayed reveal would serve stale plaintext against a destroyed key.

Arguments are hashed and never stored, so a journal cannot outlive the erasure it belonged to.

Retention, and why a wedge is urgent

A sweeper runs hourly and deletes completed invocations older than [retention] effect_journal_days (default 7), bounded by the effect’s persisted watermark. Positions above it are exactly what the next boot would replay, and reclaiming them would let their side effects fire twice.

An effect that has never persisted a watermark is never swept at all.

Getting past a wedge

1. Look. pinning_key names the lane holding the watermark down, pinning_position is where. Read last_error, then the journaled calls at GET /admin/effects/{Name}/invocations/{pinning_position}. The first missing call is where it is stuck, which is why that list pages. stuck_lanes in the same body is every other lane that is stuck, each with its own error, which is how you tell one bad customer from a dead endpoint: one entry is the former, and every entry failing alike is the latter.
2. Fix the cause and restart. The running invocation replays, and every completed call comes back from the journal instead of firing again.
3. Skip it, if the event is genuinely unprocessable:

   curl -X POST localhost:8080/effects/NotifyCustomer/skip/412

   Honoured only for a position that has already failed. Several can be pending at once, one per wedged lane, so a position out of stuck_lanes is skippable without clearing the pinning lane first: only pinning_position releases the watermark, and the others release their own lanes. A request the watermark passes is forgotten. Nothing is ever skipped automatically.
4. Erase the subject a reveal needs. That turns the failure terminal, which also unwedges it.

Terminal failures

Two things complete a position instead of retrying it: the author’s own fail "...", and a reveal of an erased subject. Both increment terminal_skips and set last_terminal_error.

That is the right behaviour for both. Neither can succeed on a later attempt.

A reveal that fails for any other reason is not terminal, and the one worth knowing is a seal whose stored text is not the document its declaration promises. That is a mismatch naming the type the declaration asked for, it retries under backoff like any failure, and it wedges the lane until the declaration and the stored seal agree again.

blocked

An arm’s @key changed while lanes were still outstanding, so the per-lane rows above the watermark are keyed under a scheme the new key never produces. last_error names the event types whose lane moved and how many lanes are outstanding.

Two ways out, and they are not equivalent:

Redeploy the previous key and let it drain. Watch until wedged_lanes and lag are both 0, then deploy the new key. This costs nothing.

hekla rewind <Effect> <watermark> against a stopped process. It discards the lane rows and the recorded invocations above the watermark, so those positions run again and perform their side effects again. An escape hatch, not the alternative.

hekla plan reports the repartition before the deploy, so this is avoidable rather than something to discover at boot.

on latest

An on latest arm runs once per key per batch, at the newest matching position in it.

Catching up, a key’s whole backlog is one invocation. It is not “skip history”: that one invocation folded the whole prefix, so it acted on the final state. latest_collapsed counts what it folded, which is why such an effect’s lag can fall by far more than its invocation count rises.

rewind

hekla rewind NotifyCustomer 0 .

Moves an effect back to a position so it reprocesses everything after it and performs those side effects again. It refuses while a server holds the data directory, prints what it would discard, and asks before doing it. --yes answers the prompt without silencing the summary; without a TTY and without --yes it refuses.

--live also lowers an on live boundary, and is permanent. It is off by default because the author of an on live arm declared that history is not news, so overriding that should be something you typed.

Rewinding to the current watermark is allowed and is not a no-op: that is the escape hatch a blocked effect needs.

There is deliberately no HTTP equivalent. An on live effect is by definition one where an accidental rewind re-sends every notification the log has ever seen.

Deploys

Graceful shutdown stops dispatching and waits up to 30 seconds for in-flight invocations, so the common deploy has nothing in flight. One abandoned by the timeout stays running and replays at the next start. An in-flight invocation whose recorded source hash differs from the code on disk is logged by name at startup.