Deadline Compensation

Problem

MaKo regulatory processes have hard SLA windows enforced by BNetzA rulings, and an inbound message starts two independent clocks:

Clock	Window	Source
Technical acknowledgement (APERAK)	45 Minuten for a UTILMD or ORDERS; Sonntag 12:00 for a Saturday arrival; nächster Werktag 12:00 for every other message type	APERAK AHB 1.0 § 2.4.1
Business answer	per Prüfidentifikator — a clock time, an end-of-Werktag, or a Werktag count	the Festlegung for that process

The two are routinely conflated, and the failure is asymmetric: a queue sized by the looser of them reports a lapsed Frist as still running.

The business windows are data, not literals — one table keyed by inbound PID, so makod, processd, obsd and agentd cannot disagree about the same deadline:

Family	Shape	Examples
GPKE Strom	wall-clock time on the n-th Werktag after the ÜT	11:00 (55001, 55077), 06:00 (55004), 05:00 (55007), 09:00 (55010), 15:00 am ÜT (55013, 55607)
WiM Strom	Werktage per PID	3 (55039) / 5 (55042) / 7 (55051) / 1 (55168)
WiM Gas	the same four windows on the Gas twins	3 (44039) / 5 (44042) / 7 (44051) / 1 (44168)
GeLi Gas	Ablauf des n-ten Werktags	4 (44001), 3 (44004/44007/44010/44016), 2 (44013)
NZR-EMob (Modell 2)	Ablauf des n-ten Werktags	7 (55238), 3 (55240, 55242)

Family names exactly these five — there is no MaBiS variant, because MaBiS publishes no answer Frist. The Prüfmitteilung in particular has none: Kap. 9.8.2 Nr. 1 leaves the cell empty and the receiving party „kann" answer, and what bounds it is the clearing window of Kap. 3.10 Tabelle 2. The two genuine 1-Werktag MaBiS obligations belong to the BIKO — forwarding a Prüfmitteilung (Kap. 9.8.2 Nr. 3) and dispatching the Datenstatus (Kap. 9.9.2 Nr. 1) — and live in mako_mabis::fristen, not in this table.

mako_fristen::antwort::antwortfrist resolves them and returns None for a PID the Festlegungen do not quantify — unknown, never unbounded. The GeLi Gas "10 Werktage" is not an answer window at all: it is the supplier's Vorlauffrist before Lieferbeginn, recorded as TEN_WERKTAGE_IS_THE_SUPPLIERS_VORLAUFFRIST because it is easy to re-introduce.

When a deadline lapses undischarged, the engine fires a DeadlineExpired event and enqueues an AperakTimeout ERP outbox message so the ERP/operator can act on the missed SLA.

Architecture

The compensation path flows through three layers:

Deadline scheduler (makod/src/orchestrator/deadline_dispatch.rs)
  └─ Process::execute_and_enqueue_with_retry(TimeoutExpired, 3)
       └─ Workflow::handle(TimeoutExpired, state)
            ├─ emit: DeadlineExpired event
            └─ outbox: AperakTimeout → OutboxErpWorker → ERP webhook

Key invariant: atomicity

execute_and_enqueue_with_retry routes through execute_command_atomic → SlateDbStore::append_with_outbox, which writes the DeadlineExpired event and the AperakTimeout outbox entry in a single WriteBatch. There is no window where:

• the event is persisted but the ERP notification is lost, or
• the ERP notification is sent but the event is missing from the audit log.

Key invariant: a deadline is discharged when its obligation is met

Deadlines fall into two kinds, and they are retired differently.

A process-response window waits on the counterparty (did they answer inside the Frist their Prüfidentifikator publishes?). It is meant to fire; Workflow::on_deadline inspects process state and returns None when the answer already arrived, which is why deadline dispatch should route through Process::execute_timeout_with_retry rather than constructing a TimeoutExpired command directly.

A delivery window waits on us (did our APERAK go out within 45 minutes?). It must never fire on the happy path, so OutboxWorker retires it the moment the message it watches is delivered — fristen::discharges_delivery_window maps each message type to the labels its delivery answers for:

Message	Discharges	Obligation
APERAK	aperak-strom-45min-window, aperak-gas-folgeprozess-…, aperak-gas-initialprozess-…	APERAK AHB 1.0 §2.4.1 / §2.3.1
CONTRL	contrl-delivery-window	CONTRL AHB 1.0 §2.3.1 / §2.4.1

A delivery discharges only its own windows — an acknowledged CONTRL says nothing about whether the application-level APERAK went out, and a deadline that merely shares the stream is left alone.

This discharge is what gives the miss counters meaning. The scheduler selects deadlines on due_at <= now, so "fired after its due time" is true of every deadline it ever hands out and proves nothing on its own. A delivery window that survives to its due time is an undelivered message — that, and only that, is the violation. A new delivery window that discharges_delivery_window does not recognise is never retired, so it alerts on every process; the every_delivery_window_label_is_discharged_by_its_message test pins that.

Retry on conflict

Deadline workers use execute_and_enqueue_with_retry(..., 3) so that a VersionConflict (concurrent event append by another task) is retried up to 3 times before bubbling to the scheduler, which re-fires the deadline later.

Workflow implementation pattern

Every workflow that registers a regulatory deadline MUST implement Workflow::on_deadline AND add compensation outbox entries in the TimeoutExpired handler:

// 1. on_deadline — map label → command (pure, no I/O)
fn on_deadline(deadline: &Deadline, state: &Self::State) -> Option<Self::Command> {
    match (deadline.label(), state) {
        ("aperak-window", SupplierChangeState::Initiated(_))
        | ("aperak-window", SupplierChangeState::ValidationPassed(_)) => {
            Some(SupplierChangeCommand::TimeoutExpired {
                deadline_id: deadline.deadline_id(),
                label:       deadline.label().into(),
            })
        }
        // Terminal or unrecognised states → no-op (idempotent)
        _ => None,
    }
}

// 2. handle(TimeoutExpired) — emit event + compensation outbox atomically
SupplierChangeCommand::TimeoutExpired { deadline_id, label } => {
    // Absorb silently on terminal states (late-firing deadline).
    if matches!(state, SupplierChangeState::Active(_) | SupplierChangeState::Rejected { .. }) {
        return Ok(WorkflowOutput::events(vec![]));
    }
    let mut outbox = vec![];
    if let Some(data) = state.initiated_data() {
        outbox.push(PendingOutbox::new(
            "AperakTimeout",
            data.new_supplier.as_str(),
            serde_json::json!({
                "pid":          data.pruefidentifikator.as_u32(),
                "malo":         data.location_id.as_str(),
                "new_supplier": data.new_supplier.as_str(),
                "deadline_label": label.as_ref(),
            }),
        ));
    }
    let event = SupplierChangeEvent::DeadlineExpired { deadline_id, label };
    if outbox.is_empty() {
        Ok(vec![event].into())
    } else {
        Ok(WorkflowOutput::with_outbox(vec![event], outbox))
    }
}

ERP delivery

The OutboxErpWorker in makod/src/core/erp_adapter.rs picks up AperakTimeout messages and maps them to ErpEventType::AperakTimeout:

"AperakTimeout" => ErpEventType::AperakTimeout,

The ERP webhook receives a CloudEvents 1.0 message with:

{
  "specversion": "1.0",
  "type": "de.mako.aperak.timeout",
  "source": "urn:mako:makod:tenant:9900357000004",
  "id": "...",
  "time": "...",
  "makopid": 55001,
  "data": {
    "malo": "DE0004...",
    "new_supplier": "9900000000001",
    "deadline_label": "aperak-window"
  }
}

Adding a new workflow with compensation

1. Add TimeoutExpired { deadline_id, label } to your XxxCommand enum.
2. Register the deadline in your workflow's Initiate handler via PendingOutbox with deliver_after derived from the regulatory Frist.
3. Implement on_deadline to return Some(TimeoutExpired) for active states.
4. Implement the TimeoutExpired arm in handle with:
   • DeadlineExpired event (for audit log).
   • AperakTimeout outbox entry (for ERP notification).
   • Early return WorkflowOutput::events(vec![]) for terminal states.
5. Add a match arm to deadline_dispatch::dispatch_deadline calling execute_and_enqueue_with_retry(..., 3). This is not optional: build_scheduler panics at startup if any workflow name an EngineModule declares through workflow_names is absent from that table, which turns the silent regulatory miss — deadline fires, falls into the unknown branch, emits a WARN nobody reads — into a deployment that will not start.

execute_timeout / execute_timeout_with_retry

Process::execute_timeout and Process::execute_timeout_with_retry are convenience wrappers that call on_deadline and route the returned command through execute_and_enqueue / execute_and_enqueue_with_retry. Prefer these in custom deadline workers over manually calling on_deadline + execute.

deadline_dispatch.rs uses both shapes, and which one is right depends on the clock. A workflow whose deadline is meant to fire — the counterparty simply did not answer — constructs the TimeoutExpired command directly, which reads plainly. A workflow where the deadline may already be discharged goes through execute_timeout_with_retry, so on_deadline gets to answer None for a settled process and no event is written at all. At present four arms take the direct form and three the on_deadline one.

A Deadline whose label no on_deadline arm matches is the silent failure this whole mechanism guards against. Workflow::on_deadline defaults to None, so an unmatched label produces a deadline that fires, does nothing, and leaves the process in its waiting state — no error, no event, no alert, and the deadline store showing it as fired. Nothing ties the string being registered to the string being matched, so makod's tests/deadline_labels.rs does: every pub const …_LABEL in a mako-* crate must appear inside some on_deadline body unless it is a delivery window, no label may be written twice (registration takes the owning crate's constant, never a literal), and no on_deadline may be a catch-all — a body that ignores deadline.label() also swallows the APERAK and CONTRL delivery windows running beside the business Frist, failing the business process because a technical acknowledgement was late.