Change safety · Hekla docs

Deploying is restarting a process over a data directory that already has history in it. Two questions matter before you do: what would change, and would it still do the same thing.

What a definition hash covers

Every declaration’s hash covers what it does, taken from the same lowered form hek digest prints. Comments, formatting and identifier positions are not in it. A const is inlined, so changing one changes every declaration that reads it.

The runtime records those hashes when it deploys, in data/hekla.db. hekla plan compares the recorded set against the candidate.

hekla plan

hekla plan .

compared 6 declaration(s) against what is deployed
  behaviour command DoA (commands/a.hk)
  behaviour command DoB (commands/b.hk)
  contract  projector UserStats (projectors/user-stats.hk)
  projector UserStats rebuilds from zero, redoing 12481 position(s)
  because `guard ShopIsConnected` changed: DoA, DoB
0 added, 0 removed, 3 changed; 1 projector(s) would rebuild

behaviour means the declaration does something different. contract means its shape changed, which for a projector is what forces a rebuild.

Beside the diff, plan counts the subject keys the boot would move under the parent their subject declares:

  3 subject key(s) would move under the parent their subject declares, at boot
    They predate the `under` that now covers them, so erasing the tenant would miss them
    until they move. `hekla adopt` does it ahead of the deploy.

That is a fact about the directory rather than a comparison of declarations, so it is counted whenever a data directory is given and a non-zero count alone means the deploy is not a no-op.

The because line is guard composition made visible: a changed guard is reported against every command that reaches it, transitively, so a one-line edit to a shared proposition does not look like a one-line change when it is not.

The rebuild line gives the cost up front: how many positions, not just which projector.

plan takes no lock, so it runs against a live deployment. It reads the event log in two cases: with --replay, and whenever the diff touches an event, record or enum declaration, where it reports what the candidate could no longer decode. Both reads are read only. It exits 0 whether or not anything would change: a change is the answer, not a fault.

What the candidate could not read

A declaration that has moved on from what is stored is the one change a hash comparison cannot price, because the cost is not a rebuild, it is a reader that stops working. So when a shape moves, plan opens the log and decodes the oldest stored event of each declared type:

  event @booking.confirmed cannot be read here (no stored event can carry `channel`), so serving would refuse to start
    A field was added to an event that already has instances, and no payload written before it can carry one: say what it reads as with `@absent(<value>)`, or make it optional so the type itself says absence is possible.

A finding that named a position came from decoding a stored event; one without came from the declaration table alone, which is the half of the check that is complete rather than sampled. serve refuses to start on exactly these findings, and the repair sentence is the same text in both places, so the two cannot come to say different things about one log. Events has the longer treatment.

`plan --replay`

This is the part no diff can do.

hekla plan --replay .

  behaviour effect NotifyCustomer (effects/notify-customer.hk)
  effect NotifyCustomer @ 4812: it reached a call the recorded run never made (http.post #0)
replayed 312 invocation(s) across 2 affected effect(s); 311 reproduce, 1 diverge
this project retains 7 day(s) of journals; anything older was reclaimed before the replay could see it
0 added, 0 removed, 1 changed; 0 projector(s) would rebuild, 1 recorded invocation(s) would diverge

Every recorded invocation of every affected effect is re-run against the candidate code, against the journal the original run left behind. Nothing is sent and nothing is mocked: the journal already holds what the world answered, so the question being asked is whether your new code would make the same calls.

--replay-limit caps how many of each effect’s most recent invocations are replayed, default 1000, minimum 1, and it requires --replay. What the cap drops is named in the report rather than dropped quietly.

An effect that reveals needs HEKLA_MASTER_KEY, and is reported as unreplayable without one rather than skipped silently.

Every one of these is named in the report rather than assumed away, which is the property that makes the coverage number worth anything. A run that ended in a terminal fail is not among them: the record cannot say whether the deployed code failed there too, so a candidate that would newly fail is reported as a divergence rather than skipped.

an erased subject
an operator skip
an invocation that journaled nothing
an unreadable record
rows already reclaimed by retention
invocations dropped by --replay-limit
an older version of the effect, excluded from the baseline rather than replayed
an effect that reads a secret this machine has not set, per effect
a reclaim that happened mid-run
a missing or wrong HEKLA_MASTER_KEY, per effect rather than per project

As a deploy gate

hekla plan --replay --json . > plan.json

Each entry in the top-level unreadable array carries event_type, a reason, a position that is null where the declaration table settled it without opening an event, and absence, true when the stored payload has no key for the field at all. absence is not the same as “add @absent”: it is also true for a sealed field, where the annotation is refused and the repair is an optional. The message’s own guidance sentence names which.

plan exits non-zero for a directory that is not a project, for error findings, for a data directory nothing was deployed to, and for a schema version this build does not expect.

A moved digest version is the one answer that is not a comparison. Every declaration’s hash carries the version of the lowered form it was taken over, so a heklang release that changes that form moves every hash at once. plan says so and compares nothing, and it still exits 0, because nothing about the project is wrong. A gate reading changes alone sees an empty diff; read digest_version_mismatch beside it.

hekla verify

plan asks about a change. verify asks whether what is already there is sound.

hekla verify .

It checks that a projector rebuilt from position 0 matches the live one, that every recorded effect invocation still replays without performing anything, and that no subject key sits under the master while its subject declares a parent:

3 key(s) of `Customer` are wrapped under the master although their subject declares a
parent, so erasing the tenant would not reach them; `hekla adopt` moves them, and
`hekla serve` does it at boot

A directory a server has opened since the parent was declared cannot be in that state, so finding it means either that it has not been served since, or that an adoption did not happen when it should have.

Quarantine applies to effects: serve --verify stops handing a diverging effect work, and /status and /admin/effects/{name} report state: quarantined with the divergence in last_error. A projector quarantines itself for a different reason, a checkpoint that moved backwards.

It takes the data-directory lock, so it refuses to run against a directory a server has open. Verify a copy of the directory, which checks the backup at the same time. Exit 0 only when the report is clean.

Continuous verification

hekla serve --verify .

Every completed effect invocation is replayed against a sealed journal as it runs. An effect that breaks the invariant is quarantined: it stops being handed work, /status and /admin/effects/{name} report state: quarantined with the divergence in last_error, and nothing clears it on its own. The record is durable, so a restart honours it rather than resuming. That is deliberate. A quarantine means the runtime found a divergence it cannot explain, and resuming automatically would be resuming into the unknown.

[verify] enabled = true in hekla.toml does the same thing, and the flag can only turn it on.

hek digest

For the narrower question of whether an edit changed anything at all:

hek digest --hash .

The digest is a deterministic rendering of what the program does with everything else taken away: const, refusal and guard are inlined and do not appear. A secret has no entry either, but it is not inlined, because there is no value in the program to inline: what appears at each read site is the name, as (secret DISCORD_WEBHOOK). Reading a different credential therefore moves the hash, and rotating a value never does. Two programs with the same hash do the same thing.

A subject is the other declaration that carries an entry of its own rather than being inlined, and it carries its under with it. The runtime files every key row under a subject’s name, so renaming one re-partitions every key, and a parent appears nowhere else in the program: without the entry, declaring one would change how every key beneath it is wrapped and move no hash at all.

Useful in CI as a cheap “is this a no-op refactor” check before running the expensive one.

A deploy, in order

hekla check . in CI, on every push.
hekla test ., likewise.
hekla secrets . against the target environment.
hekla plan --replay --json . against the live data directory, and read both divergences and unreadable. A non-empty unreadable is the one that fails at the next start rather than at the next call, so it gates the deploy on its own.
hekla adopt . against the live data directory, if a subject gained a parent. The boot does this anyway and refuses to serve without it; running it first means the deploy is not waiting on a scan of the log.
Stop the process, copy the source tree, start it.
Watch /status until projectors are ready and effects are healthy.

Step 4 is the one that is unusual, and it is the one worth keeping.