Operations

Scheduling archival, the system tables an operator opens during an incident, metrics worth alerting on, schema evolution, and the failure matrix.

Scheduling

let handle = store.maintenance()
    .interval(Duration::minutes(15))
    .expire_snapshots(false)      // snapshot retention is a compliance decision
    .spawn();

Nothing runs until you call this. A store that started a background loop on construction would surprise a process that only wanted to read.

One cycle archives every due window (bounded, so a store that has been down for a month catches up over several cycles rather than holding one process for hours), optionally expires snapshots, checks the invariant, and — where a retention policy is configured — runs the § 60 Abs. 6 sweep. In that order, so each step sees the state the cycle produced rather than the one it started from.

The handle owns the loop: dropping it stops the loop after the cycle in flight, exactly as shutdown() does, so a handle that goes out of scope cannot leave a task archiving against a store nobody is watching.

Every replica can run the same schedule. One wins the archive lease; the others report lease_contended and stop — including stopping short of snapshot expiry, which is a metadata mutation and belongs to the winner. Reading system.tables still happens everywhere, because it mutates nothing and an invariant violation is worth noticing from wherever it is seen. Contention is not a failure. Nor is deferred, the other benign no-op — see locks.

One loop, however many tables. A catalogue maintains all of its tables from one schedule:

let handle = catalog.maintenance().spawn();

Each table keeps its own watermark, archiver and lease; only the scheduling is shared. Tables are visited one after another, because twenty archivals at once turns a background job into a load spike on the database it exists to relieve.

The outcome carries one row per table, so an alert names the table rather than reporting a number nobody can act on:

for table in outcome.unhealthy() {
    tracing::error!(table = %table.table, violations = table.invariant_violations);
}

The retention sweep

The third job, and the only one that is a duty on a clock rather than an answer to a request. § 60 Abs. 6 MsbG obliges the Messstellenbetreiber to erase or anonymise personenbezogene Messwerte “spätestens jedoch nach drei Jahren ab dem Schluss des Kalenderjahres”. Nobody asks; it comes due anyway.

let handle = catalog.maintenance()
    .anonymise_after(Retention::CalendarYears(3), "§ 60 Abs. 6 MsbG", "retention-job")
    .spawn();

Off by default, for the reason snapshot expiry is: destroying a linkage is irreversible, so turning it on is the compliance decision.

CalendarYears(3) is not now - 3 years. The clock starts at the Schluss des Kalenderjahres, so a value collected on 2 January 2025 comes due on 31 December 2028; the rolling spelling would erase it a year early. Retention::Rolling(d) covers the earlier “no longer necessary” trigger.

It reads no table. What comes due is a (subject, collection year) pair, and the year is recorded on the mapping row — so the sweep is one indexed DELETE against the registry: no scan, no dependence on which rows a session can see, and the same answer through any handle. See Privacy and retention. A failed sweep appears in outcome.failures() under the name <retention>, so an alert needs no second place to look.

A failing table does not end the cycle. It becomes a row with a failure and the rest are still maintained. What fails here persists until an operator acts — a quarantined schema, an unreachable catalogue — so stopping would let one such table freeze archival for every other, whose hot tier then grows without bound for the length of the incident. healthy() is false while any table failed.

The clock is a parameter, never read. Every archival, maintenance and status call takes now, so a test drives months in milliseconds and nothing depends on when it ran. Tiering itself never reads a clock at all — it routes on from, a data value — so the only thing wall time decides is when a window becomes eligible.

Locks, and why DDL gives up

PostgreSQL grants locks in arrival order. A statement waiting for an ACCESS EXCLUSIVE lock therefore blocks every reader and writer that arrives behind it, whether or not those would have conflicted with each other. One long-running query on the hot table, or one session left idle in a transaction, turns a background job into a total ingest outage that lasts as long as the query does.

MeterStore’s hot tier issues DDL on two schedules an operator does not choose: partition creation on the write path, and partition detach and drop in the archival loop. So both are made unable to do that, in two different ways.

Creation avoids the lock

CREATE TABLE … PARTITION OF takes ACCESS EXCLUSIVE on the parent. MeterStore does not use it. It builds the relation standalone, adds the bound CHECK that lets the attach prove the partition constraint from the catalogue instead of scanning, attaches it — SHARE UPDATE EXCLUSIVE on the parent from PostgreSQL 12, which conflicts with no read and no write — and drops the now-redundant CHECK again so no row is ever checked against it twice.

The integrity constraints go on before the attach, while the relation is still invisible to every other session. A GiST index built on a table nobody can see cannot contend with anything.

Detach cannot, so it declines to wait

Removing a partition from a table’s inheritance genuinely needs the strong lock. Every DDL statement therefore runs under a lock_timeout:

[hot]
ddl_lock_timeout = "3s"    # 0s disables it — PostgreSQL's default, and the advice against

A detach that cannot get its lock promptly gives up having changed nothing. The cycle reports deferred and the next one tries again. Raise the timeout on a deployment that reports out of the same tables it writes; every second added is a second the whole table can stall for.

No drop follows the cold commit at all, so there is no second place a timeout can change state: the partition is always left detached and reclaimed on a later cycle — see the reader grace. A reclamation that cannot get its lock skips that partition and leaves the rest of the cycle running, because the only cost of keeping it one cycle longer is disk.

Alert on watermark lag, not on deferral (outcome.tables[..].deferred()). A cycle deferring once means a long query was in flight, which is ordinary. A table deferring every cycle for an hour means something holds a conflicting lock permanently — look for it in pg_stat_activity, usually a session idle in a transaction.

Errors, and which to retry

Error::is_retryable() is the split: a lost connection and an unavailable lock are worth retrying; a refused delivery, an invalid configuration and a statement that will not plan are not, and retrying them loops on a message that will never change.

The last two are the pair worth keeping straight, and the difference is who is at fault. IntegrityViolation means the store stopped something from becoming true — an overlapping delivery, two network operators for one reading, a value restated under an existing version. InvariantViolated means something already is true that should not be. Whoever is paged for the second must not be woken by a producer sending a bad row.

System tables

SELECT "table", watermark, watermark_lag_seconds,
       hot_partitions, partitions_ahead, invariant_violations, healthy
FROM system.tables;

healthy covers the two ways a table stops working, and they fail differently:

• invariant_violations — wrong answers now. Rows below the watermark still in PostgreSQL. The tier split assumes a row’s interval start decides where it lives, so anything else means a query can return the wrong number. This is the alert.

• partitions_ahead — no answers shortly. Partitions that can still hold a row written now or later. Reaching zero makes the next insert fail outright. Counted from the partitions that exist, not derived from settlement_lag + headroom: that figure is a constant, so it reports the runway a healthy deployment would have and cannot warn you about the one that stopped.

  A table with no partitions at all is exempt, because “not started” and “exhausted” show the same two numbers and mean opposite things. A table created a moment ago has no frontier to run out of, and reporting it degraded made the first status of every new deployment an alarm — which is how an alert stops being read. A store that cannot enumerate its partitions reports -1 and is not called healthy: “cannot say” must not read as “fine”.

Three more relations answer questions that otherwise need Iceberg metadata by hand:

SELECT * FROM system.config;                                       -- settings that interact
SELECT value FROM system.resolution WHERE setting = 'resolution_sql';
SELECT value FROM system.resolution WHERE setting = 'balancing_day_column';
SELECT snapshot_id, committed_at, watermark FROM system.snapshots;

The system schema is not a stored table. These are in-memory relations registered into the DataFusion session, computed when refresh_system_tables is called and refreshed only by calling it again — deliberately, so a query never silently pays for a round trip to both tiers. They live in the MeterStore process: nothing in PostgreSQL or in the Iceberg warehouse corresponds to them, and an external engine pointed at the warehouse cannot see them.

system.resolution carries the two rules an engine reading the Iceberg files directly needs and cannot derive: version resolution, without which a corrected interval is counted twice, and the balancing day, without which a gas Lastgang is grouped six hours out of phase. The first is delivered as SQL to paste; the second as the name of a column, because the Gastag has no portable SQL — so the rule is applied at write time and every engine just groups on the answer. Both are explained in full.

system.config exists because the settings that matter interact: a settlement_lag shorter than an archival_step is valid alone and wrong beside it — a window closes while corrections for it are still arriving, and they land below the watermark where no query looks. Seeing the two side by side is how that gets noticed.

All four are snapshots computed when asked. store.refresh_system_tables(now) recomputes them — deliberately explicit, so an ordinary query never silently pays for a round trip to both tiers.

Metrics

Instruments are created against the OpenTelemetry API, not an SDK. Until your application installs a meter provider, recording is a no-op. That is the right contract for a library: MeterStore decides what is worth measuring, the application decides where measurements go.

Every instrument carries a table attribute; scan metrics add tier. The retention counter carries neither — a subject’s linkage is destroyed across the whole deployment at once, so attributing it to a table would invite a sum that double-counts it.

Schema evolution

Metering schemas are regulator-defined and move on multi-year cycles, so this is far smaller than in a general framework — but not zero.

MeterStore detects schema drift; it does not perform schema evolution. There is no ALTER-equivalent in the crate: it compares the schema it is configured to write against the one the cold table actually has, before every archival run, and either proceeds or halts. Applying a change is an operator’s job, out of band, with the same Iceberg tooling that does compaction — and this table says which changes it will then accept.

That last row is worth reading twice. Iceberg resolves columns by field id, so a rename applied through Iceberg’s own UpdateSchema costs nothing and historical files keep reading. MeterStore compares by name, because a name is what the encoder writes and what an external engine queries — so a rename is invisible to it and arrives as a drop plus an add. For a nullable attribute column both halves are safe and it passes; for a non-nullable one — every identity column is one — both halves are unsafe and the table halts, which is right rather than incidental, because renaming an identity column changes the merge key.

Quarantine is the honest response to a change that cannot be applied safely. The table halts, its watermark freezes, an operator resolves it. Freezing is the point rather than a side effect: the rows stay in PostgreSQL, where they can still be corrected, instead of being archived into a layout nobody has agreed on. Silent corruption is never traded for uptime.

Three details:

• A decimal’s precision may widen; its scale may not. Precision adds representable digits; changing scale reinterprets every stored integer by a factor of ten. In a settlement figure that is a silent order-of-magnitude error.
• Both directions of a merge-key change quarantine. An identity column is non-nullable, so declaring one arrives as a NOT NULL addition and undeclaring one as a dropped required column. The second is the more dangerous: the key narrows, two readings the wider key kept apart start competing in resolution, and one supersedes the other with no error anywhere.
• A cold store that cannot report its schema is not treated as compatible. The check simply did not run, and saying so beats implying it passed.

The comparison runs at build and before every archival run. The second is not redundant: the cold table can change out of band — an operator running compaction with Spark, or a second deployment on an older configuration.

Failure matrix

Degrade, don’t lie is the rule behind that table. When Iceberg is unreachable, cold queries fail loudly rather than silently returning only the hot window.

Compaction and orphan files

Both run out of band, with Spark or PyIceberg against the same standard table — iceberg-rust can neither commit a snapshot that removes files (no rewrite action, and TableCommit’s builder is crate-private) nor list a warehouse to find files the manifests do not reference. The interop suite checks that a foreign engine reads the schema, the partition spec, the format version and the tiering watermark out of these tables.

Neither costs correctness, and compaction does not recover version elision, which is worth saying because it reads as if it should: a corrected reading has two versions stored and appears twice however the bytes are arranged. Coarser files in fact push elision the wrong way, since a scan reads whole files. The case for compaction is the ordinary one — less manifest to plan against.

The one orphan source that is closed needs no listing: a commit that wrote its data files and failed to land still holds every path, so it deletes them before returning the error. It re-reads the table first, because a commit can fail after landing.

Snapshot expiry

Ten years by default, and opt-in: a snapshot is what makes a past settlement reproducible, so retention is a compliance decision rather than a disk-space one.

It reclaims metadata, not readings. The table is append-only, so every data file an old snapshot referenced is still referenced by the current one; what expiry bounds is the metadata JSON, whose snapshot array grows with every commit and is parsed on every table load. Unreferenced manifest lists and old metadata files stay on object storage — removing those needs the listing operation above.

The retention window is meterstore’s, not the table’s

snapshot_retention and min_snapshots_to_keep decide what expiry removes, and nothing else does — including history.expire.* on the Iceberg table, which iceberg’s expire action would otherwise apply on top. Its age path runs whether or not snapshot ids are named, defaulting to max-snapshot-age-ms of five days, so a store could stop being able to reproduce a settlement older than a working week.

MeterStore pins that cutoff to the epoch, which selects nothing: the ids computed against the configured retention are the whole of what is expired. Since history.expire.* is a table property, that also keeps out-of-band compaction from deciding a deployment’s retention by setting one.

Do not expire snapshots from a foreign tool

A foreign commit carries no watermark, so the boundary lookup walks back the parent chain to find one — and removing any ancestor on that walk strands it. Not just the snapshot carrying the boundary: an intermediate one leaves a hole, the walk stops at a parent id that no longer resolves, and every query fails at once on a table that is otherwise healthy.

expire_snapshots re-stamps the boundary onto the current snapshot first, so there is no chain to punch a hole in. store.reassert_watermark() is that step on its own — run it after any out-of-band maintenance if you are not also running expiry. It republishes what the history already says, cannot move the boundary, and is a no-op when the current snapshot carries one:

// After compacting with Spark or PyIceberg:
store.reassert_watermark().await?;

Removing data

store.purge_table(name) destroys the PostgreSQL table with every partition, the Iceberg catalogue entry, and the data files in object storage. It is the only operation in the crate that deletes stored readings — everything else is append-only, because a settlement must stay reproducible.

The name must be repeated, because a handle carries no visual indication of which table it points at and there is no recovery path.

There is no way to expire part of a table’s history: removing a subset means rewriting files. For the obligation that usually prompts the question, see Privacy — the statute asks for the personal link to go, not the rows, and that is O(1).

Dropping a hot partition, by contrast, destroys nothing: it runs only after those rows are durable in Iceberg, and a partition the watermark does not cover raises an invariant violation rather than being dropped.