Signal Processing & Triggers

Overview

Signal triggers are continuous monitoring rules that evaluate conditions against live time-series data and fire alerts when thresholds are breached. They bridge the gap between passive data storage and active incident response.

Trigger Definition

Triggers are defined using a SQL-like DSL:

CREATE TRIGGER high_cpu
ON metrics
WHEN value > 90.0 AND metric_name = 'cpu_utilization'
DELIVER webhook('https://alerts.example.com/hook')
COOLDOWN INTERVAL '120s';

Components

Quoted Identifiers

Trigger and measurement names containing special characters (hyphens, dots, spaces) can be enclosed in double quotes or backticks:

CREATE TRIGGER "high-cpu.alert" ON "system.cpu" WHEN value > 90.0;
CREATE TRIGGER `my trigger` ON `my measurement` WHEN value > 1.0;

Double quotes support escaped quotes via doubling: "say""hello" produces the identifier say"hello.

Evaluation Semantics

Window Functions

Triggers evaluate sliding window aggregations against thresholds:

Evaluation Frequency

Triggers are evaluated at each new data point (push-based, driven by the streaming CDC layer). This provides sub-second latency from data arrival to alert firing.

Cooldown Mechanism

Without cooldown, a sustained threshold breach produces a flood of duplicate alerts. The cooldown period suppresses repeated firings:

Condition breached:  ●────●────●────●────●────●────●
                     ↑         ↑                   ↑
                  FIRE      SUPPRESSED           FIRE
                     ├── cooldown ──┤         (cooldown expired)

Implementation

A firing is suppressed only when both clocks agree it is inside the window: the data's own timestamps (current_ts - last_fired_ts in [0, cooldown)) and the wall clock since the last fire. Either alone is wrong — out-of-order data has a negative delta and a backfill has a huge positive one.

State is per (trigger, series), so one noisy device does not silence the others.

Alert Severity Levels

Actions

Webhook

Sends an HTTP POST with a JSON payload:

{
  "trigger": "high_cpu",
  "severity": "critical",
  "value": 95.3,
  "threshold": 90.0,
  "timestamp": "2024-01-15T14:30:00Z",
  "tags": { "host": "web-01", "region": "us-east" }
}

Composite Triggers

Triggers can reference other triggers for complex logic:

CREATE TRIGGER memory_pressure_with_high_cpu
ON metrics
WHEN value > 90.0 AND anomaly_score > 3.0
DELIVER webhook('https://alerts.example.com/hook');

Flap Detection

Metrics that oscillate around a threshold cause rapid fire/clear cycles (flapping). Chronix implements hysteresis to prevent this:

• Fire threshold: 90%
• Clear threshold: 80%

The trigger fires when value crosses 90% and does not clear until value drops below 80%. This creates a dead zone where the trigger state is sticky.

100% ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
 90% ─ ─ ─ FIRE ─ ─ ─ ─ ─ ─
      ═══════════════════════  ← sticky zone
 80% ─ ─ ─ ─ ─ ─ CLEAR ─ ─ ─
  0% ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

Performance

Triggers are evaluated in the hot path of ingestion, so they must be extremely lightweight. Window buffers are stored in a ring buffer to avoid allocations.

Signal Event Metadata

Each SignalEvent carries condition-specific metadata that provides context for downstream consumers (dashboards, webhooks, runbooks):

Deviation Mode

The ForecastDeviation trigger computes relative deviation when the forecast is sufficiently large, but falls back to absolute comparison when the forecast is near zero (to avoid division by near-zero). The deviation_mode field in the event metadata indicates which path was taken: "relative" or "absolute_fallback". This transparency ensures operators can distinguish between the two semantics.