Operating

Running Bergman as a cron job, wiring credentials, reading the audit trail, and what the exit codes mean.

bergman run performs one cycle and exits, so it fits a scheduler you already run.

As a cron job

bergman run \
  --config /etc/bergman/bergman.toml \
  --audit-log /var/log/bergman.jsonl

Kubernetes

apiVersion: batch/v1
kind: CronJob
metadata:
  name: bergman
spec:
  schedule: "0 * * * *"
  concurrencyPolicy: Forbid      # a cycle that overruns must not overlap itself
  jobTemplate:
    spec:
      backoffLimit: 0            # a failed cycle is retried by the next schedule,
      template:                  # not immediately — the table is probably busy
        spec:
          restartPolicy: Never
          serviceAccountName: bergman
          containers:
            - name: bergman
              image: ghcr.io/hupe1980/bergman:latest
              args: ["run", "--config", "/etc/bergman/bergman.toml"]
              env:
                - name: BERGMAN_CATALOG_TOKEN
                  valueFrom:
                    secretKeyRef: { name: bergman-catalog, key: token }
                - name: BERGMAN_LOG
                  value: info
              volumeMounts:
                - name: config
                  mountPath: /etc/bergman
              resources:
                requests: { cpu: 500m, memory: 512Mi }
                # See "Sizing" below: compaction's memory pool is
                # max_parallel_tables x max_sort_memory, 4 GiB by default.
                limits:   { memory: 6Gi }
          volumes:
            - name: config
              configMap: { name: bergman-config }

Overlapping cycles are safe — commits are compare-and-swap — but waste work, hence concurrencyPolicy: Forbid.

As a daemon

The daemon is a long-lived process that follows the schedules your rules declare:

bergman daemon --interval 1h

It sleeps until the earliest trigger fires. What wakes it decides what the cycle covers:

Woken byEvaluates
--intervalEvery table the catalogs hold
A rule's scheduleOnly the tables that rule's pattern matches
An eventOnly the tables it was told about
[[rules]]
match    = "prod.streaming.**"
schedule = "*/15 * * * *"      # wakes every 15 minutes, for these tables only

[[rules]]
match    = "prod.archive.**"
schedule = "0 3 * * *"         # nightly, for these
# As a Deployment rather than a CronJob, when you want the metrics endpoint.
containers:
  - name: bergman
    args: ["daemon", "--interval", "1h", "--metrics-addr", "0.0.0.0:9090"]
    ports:
      - { name: metrics, containerPort: 9090 }
    livenessProbe:
      httpGet: { path: /health, port: metrics }

Reacting to commits

The daemon can be told which table changed:

bergman daemon --listen 0.0.0.0:9090 --events --debounce 30s
curl -XPOST localhost:9090/events -H 'content-type: application/json' \
  -d '{"catalog":"prod","namespace":["analytics"],"table":"events"}'

An event cycle plans only the tables it names. Notifications within the debounce window collapse into one cycle. Timers still fire: --interval is an absolute deadline that only an interval cycle advances, so no stream of events can postpone the periodic sweep.

Bergman carries no NATS or Kafka client; bridge from the bus you run, forwarding what it receives.

Lakekeeper's CloudEvents

/events also accepts Lakekeeper's CloudEvents, forwarded unchanged:

  • Structured: a JSON body carrying specversion, sent as application/cloudevents+json.
  • Binary: the attributes as ce-* headers.

The table comes from the extension attributes name and namespace (a multi-part namespace joined by the 0x1F unit separator), and tabular-type must be table. updateTable, createTable and registerTable queue the table; any other type, and any view event, is answered 202 ignored.

A CloudEvent names Lakekeeper's warehouse, not a Bergman catalog, so the bridge names the catalog in the query string:

curl -XPOST 'localhost:9090/events?catalog=prod' \
  -H 'content-type: application/cloudevents+json' -d @event.json

Without ?catalog=, a CloudEvent is answered 400. Other envelope fields are ignored. The token, when configured, is checked before the body is read.

Authenticate it

Unlike /metrics and /health, /events causes work — listing object storage, rewriting data.

BERGMAN_EVENTS_TOKEN=$(openssl rand -hex 32) \
  bergman daemon --listen 0.0.0.0:9090 --events \
                 --events-token-env BERGMAN_EVENTS_TOKEN
curl -XPOST localhost:9090/events \
  -H "authorization: Bearer $BERGMAN_EVENTS_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"catalog":"prod","namespace":["analytics"],"table":"events"}'

The flag names a variable, never a value, and the token is compared in constant time. An unauthorized request gets a bare 401 before the body is examined. The daemon warns when /events is served without a token on an address that is neither loopback nor behind an authenticating mesh.

Asking it what it is doing

A daemon started with --listen serves /status: the cycle in progress or the next one due, how the last cycle went, and the latest outcome for every table that had work or a note. bergman status reads it and puts what needs you first:

$ bergman status --daemon http://bergman.maintenance:9090
daemon up 3h, 14 cycles finished; next at 11:00 UTC (interval, in 18m)
last cycle 14 (2 changed tables) at 10:41 UTC: 2 tables, 3 operations succeeded in 12s

needs attention
  prod.sales.orders  (cycle 12, 09:02 UTC)
    [XX] remove-orphans: vetoed by observer

maintained
  prod.sales.events  (cycle 14, 10:41 UTC)
    [ok] expire-snapshots: 8 snapshots expired

1 table needs attention.

It needs no configuration file and no credentials, so it works from a laptop pointed at a pod (BERGMAN_DAEMON_URL sets the address), and --format json prints exactly what /status serves. It exits 2 when an operation failed or was refused, or the last cycle failed outright — the same contract as bergman run — so it doubles as a check.

A table leaves the list only when a cycle that examined it had nothing to say. An event for one table, or a cycle the byte budget stopped short of it, leaves what an earlier cycle said in place, with the cycle that said it. The board lives in memory: a restarted daemon starts empty and fills in on its first cycle.

/status names tables and carries the text of refusals and errors, the same content the daemon prints to its own output. It causes no work. Expose it where you would let someone read the daemon's logs.

Windows

With a maintenance_window set, the daemon sleeps to the window's edge. SIGTERM and Ctrl-C stop it after the cycle in hand; a cycle killed outright leaves only unreferenced files, which the orphan scanner reclaims. A failed cycle is logged with its trigger and does not stop the daemon.

Prefer run under a scheduler you already operate: a CronJob gives you retries, history, alerting and resource limits the daemon does not.

Sizing

Metadata-only maintenance — expire, manifests, orphans — holds only manifests.

Compaction needs headroom. compaction.max_sort_memory (1 GiB) is the memory pool per file group, groups within a table run one at a time, and limits.max_parallel_tables (4) tables run at once:

max_parallel_tables x max_sort_memory   +   headroom
        4           x      1 GiB        +   ~1 GiB    =  ~5 GiB

Below the pool size the sort and anti-join spill to disk rather than fail, so give the container writable scratch space if you tighten it.

limits.compaction_parallelism does not enter that budget — the pool is shared across a group's threads. On a CPU-limited container, pin it to the quota:

# 2 CPUs requested in Kubernetes
[limits]
compaction_parallelism = 2

Exit codes

CodeMeaning
0Everything that ran, ran. Includes no-ops and conflicts
1Bergman could not start: bad config, unreachable catalog
2The run completed, but an operation failed or was refused

Conflicts do not set the exit code: losing a commit to a foreground writer is the design working.

Credentials

In order of preference:

Catalog-vended. If your catalog vends storage credentials — Rustberg, Polaris, Lakekeeper, S3 Tables — Bergman holds no storage secret; it receives short-lived, prefix-scoped credentials per table.

Provider chain. Instance roles, IRSA, workload identity.

OAuth2 client credentials. credential = "client-id:secret" in [catalogs.properties], optionally with oauth2-server-uri and scope — the properties every Iceberg client reads. The catalog read client and Bergman's commit client present the token of one token source, which renews on demand within a minute of expiry; concurrent requests wait for one renewal. There is no timer — see Long-lived processes.

Environment. token_env names the variable holding a static bearer token. Bergman does not renew a token it was handed.

Last resort. Keys in [catalogs.properties]. The file is then a secret.

Secrets never reach a log

Credential, the cached bearer token and CatalogConfig redact in Debug, so tracing::debug!(?bergman) in an embedding service logs no secret. Property names survive, and non-secret settings such as s3.endpoint and s3.region are shown in full:

CatalogConfig { name: "prod", uri: "https://polaris.example.com/api/catalog",
  token_env: Some("BERGMAN_CATALOG_TOKEN"),
  properties: {"s3.endpoint": "https://minio:9000", "s3.region": "eu-central-1",
               "s3.secret-access-key": "<redacted>"} }

A property is redacted when its name contains secret, key, token, credential, password or sas.

Least privilege

Bergman needs, on the warehouse prefix:

ActionNeeded for
s3:ListBucketOrphan scanning — the listing is the operation
s3:GetObjectReading metadata, and reading data files during compaction
s3:PutObjectWriting rewritten data files, manifests and manifest lists
s3:DeleteObjectOnly if you enable orphan deletion or snapshots.delete_files

inspect and plan alone need only the first two.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::lake",
      "Condition": { "StringLike": { "s3:prefix": ["warehouse/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::lake/warehouse/*"
    }
  ]
}

Grant DeleteObject only after reading a dry run and enabling deletion deliberately.

The audit trail

bergman run --audit-log /var/log/bergman.jsonl

Append-only JSON Lines, flushed per record:

{"at":"2026-08-23T02:14:07Z","run_id":"6f2c…","table":"prod.analytics.events",
 "operation":"expire-snapshots","matched_rule":"prod.analytics.*",
 "reason":"oldest snapshot is 34d old (> 7d), 61 snapshots retained",
 "result":{"result":"succeeded","detail":"58 snapshots expired, 12043 files deleted"}}

Every record names the rule that matched and the measurement that fired. Deletion manifests carry the complete file list and are written before the first delete.

# What was deleted last night?
jq -r 'select(.deleted_files) | .deleted_files[]' /var/log/bergman.jsonl

# What needs attention?
jq -c 'select(.result.result == "failed" or .result.result == "refused")' /var/log/bergman.jsonl

# How often are we losing to writers?
jq -r 'select(.result.result == "conflicted") | .table' /var/log/bergman.jsonl | sort | uniq -c

A table that conflicts every cycle is written harder than your window allows — move the window or narrow the rule.

Rolling it out

  1. inspect for a week. --table <glob> scopes the examination itself, so narrowing to one namespace reads only that namespace's metadata.
  2. plan with the policy you intend, and read the reasons and the notes: a note: line is work your policy asked for that the table cannot receive — an encrypted table, a partition under a superseded spec.
  3. run with metadata-only expiration (the default). Nothing is deleted.
  4. Orphans in dry-run. Read the counts. A table with 480 live files reporting 6,000 orphans has a configuration problem.
  5. mode = "delete", on one namespace, with --audit-log, and read the manifest afterwards.

Do not skip step 4.

Logs

The library emits tracing spans and installs no subscriber; the binary sets one up from --log/BERGMAN_LOG, writing to stderr so --format json on stdout stays machine-readable.

BERGMAN_LOG=info bergman run
BERGMAN_LOG=bergman::ops=debug bergman run   # per-module filtering

Metrics

Built with the metrics feature and served by the daemon:

bergman daemon --metrics-addr 0.0.0.0:9090
MetricLabelsWhat it is
bergman_operationscatalog, namespace, operation, outcomeOperations, counted by how they ended
bergman_operation_duration_secondscatalog, namespace, operation, outcomeHow long each took
bergman_files_deletion_announcedcatalog, namespace, operationFiles a deletion was announced for, counted before the first delete

outcome is succeeded, no-op, refused, conflicted or failed. Only failed and refused need attention; conflicted is maintenance yielding to a writer.

# Something needs looking at.
sum by (namespace, operation) (rate(bergman_operations{outcome=~"failed|refused"}[1h])) > 0

# Losing to writers repeatedly — the window is probably wrong.
sum by (namespace) (rate(bergman_operations{outcome="conflicted"}[6h])) > 0.5

There is no table label, and no rule label: a series per table would overwhelm Prometheus on a large warehouse. Group by namespace. Namespaces are capped too — the first 128 a process sees keep their own series, the rest share namespace="<over-cardinality-cap>", reported once in the log. Per-table facts are in the audit trail:

jq -r 'select(.result.result == "failed") | "\(.table)\t\(.result.error)"' \
  /var/log/bergman.jsonl

Metrics are recorded whether or not an endpoint serves them, so adding --metrics-addr later gives history from that moment.

/health reports liveness, not readiness: it does not check the catalog, so a brief catalog outage does not get a working process restarted.

For anything else, implement a MaintenanceObserver.