Library

Embedding Bergman: the API contract, the observer hook, and what the library deliberately refuses to do.

The bergman binary is a thin consumer of the same public API — anything the CLI can do, an embedder can do.

cargo add bergman --no-default-features --features catalog-rest,storage-s3

What each feature costs you

FeatureDefaultCarries
cli✅The binary, terminal rendering, logging setup, signal handling
catalog-rest✅The Iceberg REST catalog client
compaction✅Data-file rewriting — and DataFusion, which applies equality deletes as a hash anti-join
storage-s3 / -gcs / -azure✅ via storage-allCloud object stores
metrics✅Prometheus metrics and the daemon's HTTP surface: /metrics, /health, /status, /events

compaction is the only operation that reads and writes data, and the only one needing a query engine (why). For metadata maintenance only — expire, manifests, orphans — leave it off, as in the cargo add above, and carry no DataFusion. Planning is feature-independent: a build without it still plans compaction, and run() reports the operation as refused, naming the feature to rebuild with.

The shape

use bergman::{Bergman, policy::Config};

let config = Config::from_path("bergman.toml")?;
let bergman = Bergman::new(config).await?;

// Reads only.
let health = bergman.inspect().await?;
let plan   = bergman.plan().await?;

// The first call that writes anything.
let report = bergman.run(&plan).await?;

Scoped siblings — inspect_matching(pattern), plan_matching(pattern), plan_tables(&[table]) — scope the examination: an excluded table is never read. To scope which operations run, filter the plan with MaintenancePlan::retain_operations (what bergman run --only calls):

let mut plan = bergman.plan().await?;
plan.retain_operations(&[OperationKind::RemoveOrphans]);
let report = bergman.run(&plan).await?;

Any subset is safe; per-table order is preserved.

A plan may carry notes: work the policy asked for that a table cannot receive, with the reason — an encrypted table, a non-Parquet one, a partition under a superseded spec. Surface them:

for (table, note) in plan.notes() {
    tracing::warn!(%table, %note);
}

Config is an ordinary Rust value; no file is needed:

use bergman::policy::{Config, Rule, TableSettings, SnapshotSettings};

let config = Config {
    catalogs: vec![catalog],
    rules: vec![Rule {
        pattern: "prod.**".into(),
        settings: TableSettings {
            snapshots: Some(SnapshotSettings {
                max_age: Some(Duration::from_secs(7 * 86400)),
                ..Default::default()
            }),
            ..Default::default()
        },
        ..Default::default()
    }],
    ..Default::default()
};

The contract

No global state. No logger, tracing subscriber or signal handler is installed, and no configuration file is read unasked. The library emits tracing spans and events.

Bring your own runtime. Every entry point is a plain async fn. Every concurrency limit is a parameter: limits.max_parallel_tables (tables in flight), limits.io_concurrency (object-store requests per operation) and limits.compaction_parallelism (CPU width of a rewrite).

No background threads, including for credentials — a token is renewed by the request that finds it due (Long-lived processes).

Planning is pure. plan() writes and deletes nothing; dry-run is stopping before run(), which executes exactly the plan it is given.

Types are serializable. TableHealth, MaintenancePlan, EffectivePolicy and RunReport implement Serialize.

Semver. Pre-1.0, breaking changes land in minor versions with a migration note in the changelog.

Observers

The extension point. Every method has a default no-op body; override only what you need.

use bergman::obs::{MaintenanceObserver, OperationContext};
use bergman::plan::OperationResult;

#[derive(Debug)]
struct Metrics(prometheus_client::registry::Registry);

#[async_trait::async_trait]
impl MaintenanceObserver for Metrics {
    async fn operation_finished(&self, ctx: OperationContext<'_>, result: &OperationResult) {
        self.record(ctx.table, ctx.kind, result);
    }
}

OperationContext carries the run id, the table, the operation, the matched rule, and the trigger's reason.

let bergman = Bergman::builder(config)
    .with_observer(Arc::new(observers))
    .build()
    .await?;

Approval gates

operation_starting returning false vetoes the operation, reported as Refused:

async fn operation_starting(&self, ctx: OperationContext<'_>) -> bool {
    ctx.kind != OperationKind::RemoveOrphans || self.approved(ctx.table).await
}

Deletion manifests

deleting_files receives the complete list before the first deletion, so a record survives a crash mid-deletion.

Combining observers

use bergman::obs::Observers;

let observers = Observers::new()
    .with(Arc::new(AuditObserver::new(JsonlSink::open("audit.jsonl")?, run_id)))
    .with(Arc::new(metrics))
    .with(Arc::new(approval_gate));

Every observer is consulted on operation_starting, without short-circuiting, and any one may veto.

Running on a schedule

Daemon wraps the same engine and calls back with each cycle's report.

use bergman::sched::{Daemon, DaemonConfig};

let daemon = Daemon::new(Arc::new(bergman), DaemonConfig {
    interval: Duration::from_secs(3600),
    max_cycles: None,
})?;

daemon
    .run(
        |cycle| tracing::info!(n = cycle.number, trigger = %cycle.trigger, "cycle done"),
        shutdown_signal(),
    )
    .await?;

shutdown is any future; it is checked before each sleep and wins over a ready timer.

Reacting to changes

Bergman owns the trigger and the debounce; the transport is yours. Whatever receives events — a NATS subscriber, a Kafka consumer, a webhook — calls notify:

use bergman::sched::channel;

let (events, stream) = channel(Duration::from_secs(30));

tokio::spawn(async move {
    while let Some(msg) = subscription.next().await {
        events.notify(table_from(msg));   // never blocks, never fails
    }
});

daemon.run_with_events(on_cycle, Some(stream), shutdown_signal()).await?;

notify returns whether the notification was queued; a full queue drops rather than blocks. Without a daemon, Bergman::plan_tables plans a named subset.

Long-lived processes

A process that outlives its catalog token needs nothing extra. Each catalog has one token source shared by both clients — the read client (iceberg-catalog-rest, through its AuthManager hook) and Bergman's commit client. It renews on demand: the first request finding the token within a minute of expiry exchanges the credential again, and concurrent requests wait for that one exchange.

loop {
    // No credential step: a due token is renewed by whichever request needs it.
    let plan = bergman.plan().await?;
    let report = bergman.run(&plan).await?;
    // ...
}

A failed renewal fails the request that needed it, carrying the token endpoint's answer; the next request retries. A static token_env token is presented as given.

CatalogConfig::connect() returns both clients together as CatalogConnection { client, committer, policy }.

Errors

One error type, which says what to do:

match bergman.run(&plan).await {
    Err(e) if e.is_replan() => { /* the table moved; rebuild the plan */ }
    Err(e) => match e.disposition() {
        Disposition::Retry    => { /* transient; back off */ }
        Disposition::Terminal => { /* will fail the same way forever */ }
        Disposition::Replan   => unreachable!(),
    },
    Ok(report) => { /* per-operation outcomes inside */ }
}

A conflict is Replan, never Retry: re-submitting a commit computed against a moved table can resurrect deleted rows — see Architecture.