Deploying

The artefact is the source tree

There is no build step, no compiled output and no compile cache. What you deploy is the .hk files, plus hekla.toml if you have one, plus the master key if the project declares a subject.

Deploy is copy and restart. There is no hot reload: the project is read at startup, and a running process never re-reads it.

The data directory

data/
  events/                tephra segments, the immutable source of truth
  projectors/{Name}.db   one read model and its checkpoint
  hekla.db               effect journals, subject keys, deployed declarations
  hekla.lock.db          the exclusive one-process lock

Two of those are irreplaceable: events/ and hekla.db. Projector databases are rebuildable from the log, so they are the part you do not need to keep.

The lock lives in its own file rather than inside hekla.db, because the operational database is written continuously and an exclusive transaction held for the process lifetime would deadlock against it.

Opening any of these files directly is not a supported surface, and the table layout is private.

Backup

Copy the directory of a stopped process.

The good pattern is to stop, copy, start, and then run hekla verify against the copy. That checks the invariants and the backup in one pass, on a machine that is not serving traffic.

Migrations

The operational database migrates itself on open, one schema version per transaction, and refuses a database newer than the build:

operational database is at schema version 12, newer than this build (11)

So a downgrade is a refusal rather than a corruption. hekla plan refuses a directory this build has not migrated yet; run hekla serve against it once first.

A key namespace written before subjects were declared

One migration needs an answer from you, and only for a directory that already holds sealed data written by hekla 0.6.0 or earlier. A key row is filed under a subject’s declared name, and a directory written before the declaration existed files it under the field an @subject annotation named. The wrapped secret carries across untouched; the label does not, and nothing in the database records which field became which subject. A row under a label the program never looks up is unreachable, and every value sealed under it reads back absent, which no caller can tell from an erasure.

So the upgrade refuses until it is told the mapping, rather than succeeding and leaving data that looks deleted:

HEKLA_V10_SUBJECTS="customer_id=Customer,shop_id=Shop" hekla serve ./shop --data-dir ./data

The refusal names every namespace it found, which is the list to answer. A namespace that should keep the spelling it has is written as itself, legacy_ref=legacy_ref, which is also what to write for one you have decided to abandon. Two namespaces may merge onto one subject, and that is refused only if both hold a key for the same id. An entry naming a namespace this directory has nothing under is a warning, so one variable can cover several directories. The right-hand side is taken on trust, because the migration runs before any program is loaded: check it against your .hk files.

A refused upgrade changes nothing, the schema version included, so the previous build still opens the directory while you work the mapping out. Unset the variable once the directory is upgraded; it is read on that one step and does nothing afterwards.

Giving a subject a parent

A deploy that adds under to a subject whose keys already exist moves every one of them under that parent before it serves, and refuses to serve if it cannot. That is a scan of the log, holding the data-directory lock with the listener down, so on a large log it is the part of the boot that takes time.

hekla adopt does the same work ahead of the deploy, against the live directory and without the lock, so the boot that follows finds nothing left to do. hekla plan counts what is waiting before either.

The event log is not migrated, it is read as it stands. An event type carries no version and there is no upcast hook. A field added to an event that already has history says what it reads as with @absent(<value>), or becomes optional. See Events.

That is checked before the process serves anything. serve compares the program’s declarations against every shape ever recorded for those types, then decodes the oldest stored event of each declared type. A field no recorded shape can answer for, or a stored payload this program can no longer read, refuses the start:

this program cannot read events that are already in the log, so it cannot serve:
`@booking.confirmed` (no stored event can carry `channel`), so nothing has been
recorded and correcting the declaration and deploying again is the whole of the repair.
A field was added to an event that already has instances, and no payload written before
it can carry one: say what it reads as with `@absent(<value>)`, or make it optional so
the type itself says absence is possible.

The message carries its own repair, one sentence per distinct fault. verify runs the same check and refuses the sweep with the same findings, saying it cannot verify rather than cannot serve, and noting that nothing was changed. Both hold the data-directory lock, so neither is something to point at a directory a server has open.

The first half of the check is complete: it is settled from the declaration table without opening an event. The second samples one stored event per declared type, so it is a floor rather than a survey, and it stays silent about a type the first half already answered for. hekla plan reports both without refusing, whenever the diff touches an event, record or enum declaration, and it is the one that takes no lock. See Change safety.

Observability

Logs

tracing, filtered by RUST_LOG, default info. Installed by serve and verify only, so every other subcommand stays plain and machine-parseable: hekla openapi . > openapi.json and hekla plan --json are safe to pipe.

Colour is on only when stdout is a terminal and nobody opted out. NO_COLOR with any non-empty value, or --no-color, turns it off.

Lines worth alerting on: a failed ctrl-c handler registration (shutdown must then be forced), a completed projector rebuild with its duration, and an in-flight invocation whose recorded source hash differs from the code on disk, logged by name at startup.

/status

The whole runtime in one document: log head, uptime, every projector’s position, lag, readiness and last error, and every effect’s state, lag, wedged lanes, pinning key and pinning position. This is what a console or a health dashboard should read.

/metrics

Prometheus text exposition. Gauges are read at scrape time, so there is no interval and no [metrics] section.

Gauges

hekla_build_info, hekla_uptime_seconds, hekla_log_head_position, hekla_module_info, hekla_projector_up, hekla_projector_position, hekla_projector_lag, hekla_projector_readiness, hekla_projector_last_progress_timestamp_seconds, hekla_effect_up, hekla_effect_position, hekla_effect_lag, hekla_effect_state, hekla_effect_wedged_lanes, hekla_effect_consecutive_failures, hekla_effect_retry_backoff_seconds, hekla_effect_last_progress_timestamp_seconds

Counters

hekla_events_appended_total, hekla_commands_total, hekla_command_refusals_total, hekla_command_conflict_retries_total, hekla_projector_events_total, hekla_projector_rebuilds_total, hekla_effect_invocations_total, hekla_effect_restarts_total, hekla_effect_terminal_skips_total, hekla_effect_live_suppressed_total, hekla_effect_collapsed_total, hekla_effect_http_requests_total, hekla_reads_total, hekla_read_waits_total

Two labelling rules are design commitments rather than conventions.

A lane key is never a label. Every label is a declaration: a command, projector, entity, effect or event name, a refusal code, or a fixed outcome word. The reason is erasure, not cardinality. A lane key is a partition key, and /status may name it because it is a live view an erasure passes through; a scrape may not, because it is a copy taken somewhere hekla erase cannot reach. So the metric counts wedged lanes and /admin/effects/{name} names them.

A refusal series appears on first use. heklang inlines a refusal, so hekla cannot enumerate the codes to prime them at boot. Any expression over hekla_command_refusals_total needs or vector(0). Every other counter reads 0 from the first scrape.

Alerts

The repository ships a rules file at docs/monitoring/hekla-alerts.yml with sixteen alerts, each carrying the reasoning for its threshold inline: HeklaDown, HeklaProjectorStalled, HeklaEffectWedged, HeklaEffectQuarantined, HeklaEffectBlocked, HeklaCommandConflictsExhausted, HeklaLogNotAdvancing, HeklaReadYourWritesTimingOut and others. Start from it rather than from scratch.

/metrics is unauthenticated like everything else, so the scrape target is reached the same way /admin is.

The limits, stated plainly

Saying these first is the point, so here they are without hedging.

Nothing is authenticated. The bind address is the boundary and it defaults to loopback. A caller who reaches the port can append events, browse the console, read metrics and skip an effect’s work.

One process per data directory, enforced by an exclusive lock. There is no clustering, no replication and no horizontal scaling.

Deploy is restart. No hot reload, no upload API, no live editing.

No version on an event type and no upcast hook. A field added later reads through @absent, but renaming one or changing its type means a new field or a new type.

A scan filters on one index. Equality across any prefix of a declared index, plus a range on the column after it, and nothing wider: a filter spanning two indexes is a 400 rather than a table scan.

No automatic dead-lettering. A wedged effect is never advanced automatically; the manual skip endpoint is the only escape hatch.

No hekla fmt and no language server. Both arrive when heklang has a language server of its own. hek fmt exists today and is what to point an editor at; see Editor setup.

/docs needs outbound network, because it loads its reference UI from a CDN. /openapi.json does not.

invoke is exactly-once only when the target command is idempotent under replay. Raw http.* is at-least-once. That is an inherent property of the design rather than future work: the journal can guarantee a call is not repeated after a crash, but it cannot make a remote endpoint idempotent for you.

Pre-1.0. hekla is 0.7 and heklang is 0.9. Both carry breaking changes between minor versions, and neither has run your workload yet.

And one that will not change

heklang is the only authoring surface, permanently. There is no Rust, TypeScript or WASM SDK, and there is not going to be one. A single pure sandboxed authoring language is what makes the rebuild, the replay and plan --replay sound, and an escape hatch into a general purpose language would take all three away.