Operations

Deploying, configuring and monitoring Chronix.

Deployment

Single-Node (Embedded)

use chronix::Chronix;

let db = Chronix::builder()
    .data_dir("/var/lib/chronix")
    .build()
    .await?;

Single-Node Server

# Install
cargo install chronixd

# Run with defaults (data in ./chronix-data, listen on 0.0.0.0:8086)
chronixd

# Custom configuration
chronixd --config /etc/chronix/chronixd.toml

Validate the file before it reaches a node. --check-config applies the file, the environment and the flags, runs every startup validation, prints what it resolved to, and exits without opening the database or binding a port — so a typo, a [tracing] section this build cannot honour, or an [auth] section that authenticates nobody fails in CI rather than on the node:

chronixd --config /etc/chronix/chronixd.toml --check-config

In a container, with the file mounted:

docker run --rm -v /etc/chronix:/etc/chronix:ro \
  ghcr.io/hupe1980/chronixd:latest \
  --config /etc/chronix/chronixd.toml --check-config

It exits non-zero on a bad file, so it works as a deploy gate. See Configuration.

Multi-Node Cluster

The distributed tier is frozen and excluded from the default build; it needs --features cluster. See Cluster.

Configuration Reference

chronixd --config /etc/chronix/chronixd.toml. Unknown keys are a startup error, not a warning: every configuration struct is deny_unknown_fields, so a typo fails loudly instead of being ignored.

Server — [server]

Every setting below lives under a [server] table. A key the server does not know is a startup error naming it.

Storage — [database]

[database]
data_dir = "/var/lib/chronix"
wal_fsync_policy = "per_batch"
compression = "zstd"
zstd_level = 3

wal_fsync_policy = "periodic_5000" coalesces fsyncs onto a background thread at that interval instead of syncing per write. On flash-backed storage the fsync rate is the wear rate, which is what the policy is for; the cost is that up to one interval of writes is lost on an unclean shutdown. Deletes are synced regardless of policy. The chronix_wal_fsync_total counter is the observable form of the setting.

Query limits

Three deadlines, three engines. sql_query_timeout_secs bounds DataFusion, prom_query_timeout_secs bounds the PromQL evaluator, and database.query_timeout_secs bounds the engine's own scan — the one the native query endpoint and every export use. A deployment that raises only the first leaves the other two at their defaults. All three are enforced inside the work, so a query that runs out of time stops rather than being abandoned while it continues.

Write limits

The JSON and Line Protocol write routes all enforce max_write_batch_size and reject an oversized request with 400. Every write path — REST, Line Protocol, OTLP, Prometheus remote write, gRPC, Flight SQL DoPut, and the Kafka and MQTT connectors — is wrapped in write_timeout_secs and answers 504 or DEADLINE_EXCEEDED when it expires.

Rate limiting

Multi-tenancy

[server]
multi_tenancy = true

Off by default. When on, every ingested point is stamped with the namespace of the request that wrote it — X-Namespace for HTTP, x-namespace metadata for gRPC and Flight SQL — and every read is confined to that namespace: REST, SQL, PromQL, the Prometheus label and series endpoints, Prometheus remote read, Flight SQL and gRPC. A client-supplied __namespace__ tag is overwritten, and the tag never appears in a result or a schema.

Decide before ingesting. Points written with tenancy off carry no namespace and are invisible to a scoped read; points written with it on are not addressable with it off. Switching a populated server is a re-ingest.

Ingestion connectors carry no request, so each takes its namespace from its own configuration:

[kafka]
namespace = "tenant-a"

Namespaces and their quotas are managed through /api/v1/namespaces and persisted to {data_dir}/namespaces.json with atomic writes, so they survive a restart.

Measurement names are still process-wide: a tenant can learn that another tenant's measurement exists, and reads no rows from it.

TLS — [tls]

[tls]
cert = "/etc/chronix/server.crt"
key = "/etc/chronix/server.key"
client_ca = "/etc/chronix/ca.crt"   # optional; enables mTLS
reload_interval_secs = 60           # default; polls the files and hot-reloads, 0 disables

Authentication and authorization

[[auth.api_keys]]
name = "ingest"
key = "$CHRONIX_INGEST_KEY"   # env-var reference, Argon2 PHC string, or plain text
namespaces = ["tenant-a"]     # required when multi_tenancy = true
admin = false                 # required for restore, namespace and key management

[auth.jwt]
# issuer, audience, and either a secret (HS*), public_key_pem_file, or jwks_url

[server]
authz_policy_dir = "/etc/chronix/policies"   # Cedar policies; absent = open mode

[audit]
path = "/var/log/chronix/audit.jsonl"
hmac_key_env = "CHRONIX_AUDIT_KEY"

auth.exempt_paths defaults to the health endpoints. Cedar is default-deny once authz_policy_dir is set — an empty directory refuses everything, which is why "no directory" means no engine rather than an empty one. Policies are validated against the compiled-in schema at startup, so a rule naming an action or entity type the server never asks about stops the start rather than loading and never firing. authz_policy_dir without an [auth] section is also a startup error: a policy names a principal.

Two settings fail open if left out, so check them before going live:

• A key with no namespaces under multi_tenancy = true reads every tenant. The server refuses to start rather than allow it.
• A key with no admin cannot reach restore, namespace management or key management. Grant it deliberately, to the one key that needs it — and with Cedar configured, grant each administrative key the one capability it needs (action == Chronix::Action::"ManageBackups") rather than the whole Admin group.

Without an [audit] section the audit trail lives only in the process log. With one it is fsynced, and its hash chain continues across restarts — a chain that restarts looks exactly like one that was truncated.

Cluster — [cluster]

Requires --features cluster; the tier is frozen and not part of the default build.

Cold archive

Archiving is an embedded API call — Chronix::archive_cold_segments behind the object-store feature — not a server setting. It reads each cold (measurement, shard) group through the read path — deduplicated, with tombstones applied — writes it to object storage as one Parquet object under measurement=<m>/shard=<n>/, verifies it, and only then removes the source segments from the hot database. The object carries the hot tier's schema, _time included, so a query moved to the archive is the same query. It is queried as its own SQL table via register_cold_tier (one table per measurement), or directly with DuckDB, Polars or Spark. See Object Storage.

Parquet export

Exports are restricted to {data_dir}/exports/. Only the filename component of a requested path is accepted; ../ traversal is rejected.

Partial Deletes

Metric: chronix_delete_segments_skipped_total counter.

A predicate delete skips segments it cannot open or read rather than aborting, so a non-zero value means matching data may still be on disk. Alert on any increase if you rely on deletes for an erasure obligation:

- alert: ChronixPartialDelete
  expr: increase(chronix_delete_segments_skipped_total[15m]) > 0
  annotations:
    summary: "A delete could not scan every segment — data may remain"

The same signal is returned inline as segments_skipped on the HTTP and gRPC delete responses, so callers can retry without waiting for a scrape.

Tombstone backlog

Metric: chronix_tombstone_gc_total counter.

A delete writes tombstones; compaction is what removes the rows from disk. A tombstone is reclaimed only once every segment it was issued against has been rewritten or deleted, so the gap between "the delete returned" and "the bytes are gone" is one compaction cycle for the affected segments. Two operational consequences:

• An erasure obligation is not discharged by the delete alone. If you must be able to say the bytes are gone, force a compaction of the affected measurement and confirm chronix_tombstone_gc_total advanced.
• A tombstone set that never shrinks means compaction is not reaching those segments — check chronix_compaction_tasks_completed_total and the L0 depth rather than the delete path.

Deletes themselves are durable as soon as they return: tombstones are appended to the catalog manifest and fsynced, not left to the data WAL, which is truncated on the next flush.

PromQL scan cache

Metrics: chronix_promql_scans_total and chronix_promql_scan_cache_hits_total, both labelled kind="instant"|"range".

A range query is meant to read its window once rather than once per step. Whether it does depends on the shape of the query, so the ratio is the signal:

- alert: ChronixPromQLScanCacheIneffective
  expr: |
    rate(chronix_promql_scan_cache_hits_total{kind="range"}[10m])
      / rate(chronix_promql_scans_total{kind="range"}[10m]) < 0.5
  annotations:
    summary: "Range queries are re-reading their window per step"

A low ratio on kind="range" points at a dashboard issuing many-step queries whose selectors miss the cache; a low ratio on kind="instant" is expected and carries no information, since an instant query has one step.

Retention and Rollups

Retention (retention_secs, and measurement_retention per measurement) is enforced by the database's own maintenance thread every maintenance_interval (30 s by default). It is opt-in: with no retention configured, nothing is ever deleted.

Age is measured from min(wall clock, newest timestamp the database holds). Two consequences:

• A clock that reads far ahead — a dead RTC, a bad NTP server, a restored VM snapshot — cannot expire data the data itself does not justify expiring.
• A database that stops receiving writes keeps its history. One fresh write moves the reference to the present and the backlog expires on the next pass.

The newest data is therefore never expired, whatever the rule says, and a measurement that stops being written stops freeing disk. Use DELETE to reclaim that space.

Retention releases what it deletes. On any pass that deleted something, the cardinality budget (max_series_cardinality, reported as chronix_series_count) and the last-value cache are re-derived from what the database still holds, so device churn does not climb towards the cardinality limit and last_value() stops answering for a series whose data has aged out. chronix_series_released_total counts the released series. An emptied measurement is still a queryable table returning no rows; it leaves the measurement list only when it is dropped.

Rollups are materialised, never computed as a by-product of compaction or retention. A bucket is aggregated once, over a deduplicated scan of its source, as soon as its input is final: for a raw measurement, once the out-of-order window (±2 shards of the newest write) has closed over it; for a tier fed by another rollup, as far as that rollup has been materialised. compact() — which the maintenance thread runs every maintenance_interval — materialises after each pass, enforce_retention() materialises before it drops, and db.materialise_rollups() is the explicit call. Definitions and watermarks live in the catalog, fsynced before the call that changed it returns and carried forward by every snapshot, so a rollup cannot be lost to a half-written file beside the data directory.

db.rollup(name, start, end) reads the tier as a view: the materialised buckets below the watermark spliced onto the unfinished ones above it, aggregated from the source with the same accumulator. Reading the target measurement directly instead ends a chart short by however much has not been materialised yet. db.rollup_where(name, start, end, &[("host", "a")]) is the same view restricted to matching series; every key must be one of the rollup's group_by_tags, because the target measurement carries only those and a filter on any other key would narrow the live half and match nothing in the materialised one.

Late writes are repaired, not ignored. A backfill or a delete into a range a rollup has already aggregated records an invalidation; the next pass deletes the stale aggregates over that range, recomputes them, and cascades the repair to any tier downstream. refresh_rollup(name, start, end) does the same immediately, for a change the engine could not have observed — an out-of-band restore, say. chronix_rollup_invalidations_total counts what has been marked, chronix_rollup_ranges_recomputed_total what has been repaired, and chronix_rollup_failures_total any rollup whose pass failed — the others still run.

Rollup-aware retention trades raw data for its aggregates, so a segment is only dropped once every tier of its rollup chain has been materialised past it and has no repair pending — for the per-measurement rules as well as the global one. A segment that is still needed is preserved and counted in chronix_retention_segments_awaiting_rollup_total and in the pass's segments_preserved; a segment that cannot be read fails the materialisation that needs it, so it is preserved too. shards_dropped counts the shards the pass removed, so a pass that dropped nothing logs Retention preserved every expired segment rather than a count. chronix_rollup_points_written_total counts what materialisation wrote. Cold archiving removes a segment from the hot database, so it waits on the same gate (chronix_cold_archive_groups_awaiting_rollup_total).

Retention runs whenever any rule is configured — global, per-measurement, or a rollup's own retention_secs.

A rollup's retention_secs (retention_ns in the Rust API) is the retention of its target measurement, applied by the same pass.

Region Auto-Split Process

Chronix uses a two-phase split to ensure zero downtime during region splits:

Phase 1 — Prepare:

1. Source region transitions to Splitting state (still accepts reads and writes)
2. Two child regions are created in Replicating state covering each half of the hash range
3. Data begins replicating to children

Phase 2 — Commit:

1. Routing table atomically updated: source removed, children promoted to Active
2. Reads and writes redirect to the new child regions
3. Old region metadata cleaned up

The split triggers when either region_size_threshold or region_series_threshold is exceeded. Monitor splits via the chronix_cluster_region_splits_total counter metric.

Monitoring

Prometheus Metrics

Chronix exposes Prometheus metrics at /metrics (default port 8086).

Cluster Health

Performance

Object Store

Segment I/O

Streaming Aggregation

Self-maintenance

Flushing, compaction, rollup materialisation and retention all run on one thread per open database. Alert on chronix_maintenance_running == 0: a process that has lost it looks healthy until the memtable fills, after which only a restart recovers. /ready answers 503 naming the condition, and a full memtable with no thread to flush it is 503 OVERLOADED rather than retryable back-pressure.

Grafana Dashboards

Pre-built Grafana dashboards are in the dashboards/ directory:

Import via Grafana UI: Dashboards → Import → Upload JSON.

Distributed Tracing

Enable OpenTelemetry trace export:

[server]
log_format = "json"

[tracing]
service_name = "chronixd"

[tracing.otlp]
endpoint = "https://otel-collector:4317"
sampling = { ratio = 0.01 }
always_sample_errors = true

Requires --features otlp; without it the section is a startup error rather than a collector that never receives anything.

always_sample_errors (default: true): When enabled, the sampler is overridden to AlwaysOn, ensuring all traces — including error-bearing ones — are captured. This is necessary because OpenTelemetry sampling decisions are made at span creation time, before errors occur. Set to false and use sampling.ratio for controlled sampling in high-throughput environments where capturing every error trace is not required.

Named spans propagated across cluster:

Backup & Restore

A backup is a checkpoint: the database as of the flush that starts it. Every write acknowledged before the request is in it; a write accepted while it runs is not. Segment files are hard-linked when the target is on the same filesystem, so a checkpoint of a large database is near instant and takes no extra space until those segments are compacted away.

The directory it produces is a complete database plus a backup_manifest.json. The files are immutable, so rsync, tar and object-store sync all work on it — which is not true of a live data directory.

Create a backup:

POST /api/v1/admin/backup
Content-Type: application/json

{
  "target_dir": "nightly-2026-09-11"
}

Response: 200 OK with a BackupManifest — version, created_at, wal_sequence, segments, file_count and total_bytes. Paths resolve inside backup_root (see Configuration).

Backing up repeatedly into one directory is a rolling checkpoint: files the new one does not name are removed.

On a schedule:

[database.checkpoints]
interval_secs = 21600                    # every six hours
directory     = "/mnt/backup/chronix"
keep          = 4                        # a day of history

The maintenance thread takes these, beside the flush, compaction, rollup and retention passes. The first runs at startup. Each writes <directory>/<timestamp>; the oldest are removed once more than keep remain, and an unfinished run is removed on sight. A failed run leaves the existing checkpoints alone.

Point directory at a different volume from data_dir if the aim is surviving that volume: on the same filesystem the segments are hard-linked, so the checkpoint is not a second copy of the bytes.

Verify a backup, without restoring it:

POST /api/v1/admin/backup/verify
Content-Type: application/json

{
  "backup_dir": "nightly-2026-09-11"
}

Response: 200 OK with the backup's BackupManifest, or 400 naming the segment that is missing or the wrong size. It reads only, so it is safe against the backups you are keeping.

Restore a backup:

POST /api/v1/admin/restore
Content-Type: application/json

{
  "backup_dir": "nightly-2026-09-11",
  "target_dir": "restored"
}

Response: 200 OK with the backup's BackupManifest.

The target directory must not exist. Every segment the backup's catalog names is checked for presence and size before anything is copied, so an incomplete backup is refused. Segments are hard-linked where possible; the catalog and WAL directories are copied. The result can be opened on any machine.

Restore does not touch the running server: point a chronixd at the restored directory, or stop the server, swap the directories and start it again.

Alerting. chronix_backup_failures_total, and increase(chronix_backups_total[1d]) == 0 if you take one nightly. Both are published at zero from startup, so either can fire from the first scrape.

Object-store durability

Cold-tiered segments carry the bucket's own durability and server-side encryption. For cross-region safety use the bucket's replication.

Admin REST API

The admin REST API under /api/v1/admin/ covers backup and restore, analytics model administration, API keys and namespaces — all of which work on a standalone server — plus cluster management, which needs a running cluster and returns 503 in standalone mode. Every endpoint requires a credential with the admin capability.

Cluster Management

Runtime Log-Level Adjustment

Adjust the tracing log level without restarting the server. Uses a tracing_subscriber::reload::Layer so changes take effect immediately across all spans.

curl -X PUT http://localhost:8086/api/v1/admin/log-level \
  -H 'Content-Type: application/json' \
  -d '{"level": "debug"}'
# {"previous": "info", "current": "debug"}

Accepted levels: trace, debug, info, warn, error. Invalid levels return 400 Bad Request.

Circuit Breaker (Write Router)

The write router includes a per-node circuit breaker to avoid sending writes to unhealthy DataNodes. When consecutive write failures to a node exceed the threshold, the circuit opens and subsequent writes to that node are short-circuited with an immediate error until the recovery timeout elapses. After the timeout, the breaker transitions to HalfOpen and admits exactly one probe request — all other callers are blocked until the probe resolves. A successful probe closes the circuit; a failed probe re-opens it.

Metric: chronix_circuit_breaker_open_total counter (labels: node_id) — incremented each time a node's breaker opens. There is no gauge of the current state; rate(...[5m]) > 0 is how you alert on a node that keeps tripping.

Analytics Model Management

Re-training clears the specified models from the catalog. The ContinuousForecastEngine will automatically re-fit them on the next incoming data batch.

Namespace Management

Namespace routing uses the X-Namespace header. If omitted, the default namespace is used. Namespace definitions are durably persisted to disk and survive process restarts.

When the data volume fills

1. A write that cannot reach the WAL answers 507 STORAGE_FULL. Clients retry; the condition usually clears.
2. The WAL rewinds to its last durable offset, so writes resume the moment space returns — no restart — and a record reported as failed is never written later.
3. A flush that cannot write its segment leaves the memtable frozen for the next attempt, so nothing acknowledged is lost.
4. Reads keep working and /ready stays 200: back-pressure, not a reason to leave the load balancer.

Fix it by freeing space — retention, WAL max_file_size, or the cold tier. Alert on chronix_write_errors_total{reason="storage_full"}.

A failed fsync is different and does stop writes: the kernel may drop the dirty pages and clear the error, so the file's contents are unknown. /ready answers 503 with the reason, and so does every write — 503 OVERLOADED, unredacted, naming the condition. Recovery is to close and reopen the database.

When ingestion outruns the storage

A memtable at capacity is back-pressure, not a fault: the flush that clears it is already running. Writes answer 503 BACKPRESSURE with Retry-After: 1 and a message naming the limit, so an agent backs off for a second and carries on. It is the ordinary condition on a gateway whose flash is slower than its ingest. Alert on chronix_write_admission_rejected_total.

OpenAPI Specification

The full REST API specification is available as an OpenAPI 3.1 JSON document:

GET /api/v1/openapi.json

This endpoint returns a machine-readable API description suitable for code generation, documentation tools (e.g., Swagger UI, Redoc), and API testing frameworks.

The servers[0].url field in the OpenAPI document is dynamically derived from the server's configured listen address and TLS settings, so generated clients automatically target the correct host and port.

Connector Management

Startup Retry

Ingestion connectors are started at boot. By default a connector that cannot reach its broker stops the server, rather than leaving it running with a silently dead ingestion path.

Raise the retry count when connectors and brokers start concurrently, as in a container stack.

Troubleshooting

Common Issues

API Key Rate Limiting

API key validation failures are rate-limited to prevent brute-force attacks.

When the failure count for a source exceeds the threshold within the sliding window, subsequent API key validation attempts are rejected with 429 Too Many Requests until the window expires.

See Also

• Cluster Operations — cluster setup, scaling, failover
• API Reference — comprehensive endpoint documentation
• Security Guide — production security checklist
• Performance Tuning — workload-specific tuning
• Analytics Guide — forecast and anomaly detection APIs
• Architecture Reference — component deep-dive
• Guide