agentd Operator Guide

agentd operator guide: Multi-agent LLM orchestration daemon. 28 built-in specialists ship in container image. Orchestrator + Specialist Mesh, LanceDB RAG, parallel dispatch, A2A agent cards, OpenAI / Anthropic / AWS Bedrock.

`agentd` — Multi-Agent LLM Orchestration

agentd is the AI automation layer for the mako platform. It connects large language models to all 16 production services via MCP, enabling automated analysis, decision support, and workflow orchestration.

Port: :9580

Endpoint	Description
`POST /webhook`	Inbound CloudEvent trigger (HMAC-verified)
`POST /api/v1/run`	Manual agent invocation (OIDC JWT required)
`GET /api/v1/sessions`	Last 100 agent decisions (in-memory ring buffer)
`GET /api/v1/dlq`	Dead-letter queue: pending retries + recent exhausted entries
`GET /api/v1/agents`	List all active agents (built-in + custom)
`GET /api/v1/agents/catalog`	Full catalog of all 28 built-in definitions
`GET /.well-known/agents/{name}`	A2A Agent Card for a specialist
`POST /api/v1/rag/ingest`	Index a live text document into LanceDB
`POST /api/v1/rag/search`	Query the RAG knowledge base directly
`GET /health` · `GET /health/ready`	Liveness / readiness

Key design decisions

28 built-in specialists ship in the container image

The most important architectural change from the naive "put prompts in demo config" approach: all 28 specialist system prompts are compiled into the agentd binary and ship in the container image. Operators activate them via [bundled_agents] in agentd.toml without copying hundreds of lines of system prompts.

This follows the same principle as makod's compiled-in AHB profiles — domain knowledge lives in the binary, not in operator-managed config files.

A2A Protocol compliance

Each specialist exposes an A2A Agent Card at /.well-known/agents/{name} — a standards-based capability declaration that enables external systems to discover and interact with mako specialists without prior configuration.

Parallel dispatch

The orchestrator supports three dispatch modes:

Mode	Behaviour	Best for
`sequential` (default)	Route to one specialist	Clear single-domain events
`parallel`	Fan out to ALL matching specialists concurrently	Compliance events needing multiple checks
`race`	Fan out; return first specialist to complete	Latency-sensitive events

Architecture

graph TB
    TRIGGER["Trigger<br/>CloudEvent webhook<br/>or POST /api/v1/run"]

    subgraph orchestrator ["Orchestrator Agent"]
        ORCH["1. Direct match (trigger_patterns)<br/>2. LLM triage (tool call)<br/>3. Fallback: orchestrator handles directly"]
        MODE["DispatchMode:<br/>sequential | parallel | race"]
    end

    subgraph builtin ["28 Built-in Specialists (compiled into binary)"]
        direction LR
        MAKO["mako-agent<br/>EDIFACT · UTILMD · deadlines"]
        BILLING["billing-agent<br/>billingd · invoicd · O2C"]
        BILREG["billing-regulatory-guard-agent<br/>§40/§41/§41a/§42 compliance"]
        JAHRB["jahresabrechnung-agent<br/>Annual settlement orchestrator"]
        ANOMALY["billing-anomaly-agent<br/>20% deviation check"]
        EEG["eeg-agent + eeg-compliance-agent<br/>einsd · §52 · §44b · §20 EEG"]
        MORE["... 19 more specialists"]
    end

    subgraph rag ["RAG Knowledge Base"]
        LANCE["LanceDB<br/>S3 / GCS / local<br/>ANN vector search"]
    end

    subgraph tools ["MCP Tools — 14 MCP services (150+ tools)"]
        T1["makod · marktd · billingd"]
        T2["edmd · accountingd · obsd"]
        T3["einsd · netzbilanzd · sperrd · ..."]
    end

    TRIGGER --> orchestrator
    orchestrator --> builtin
    builtin -->|"ReAct: reason → act → observe"| tools
    builtin -->|"background knowledge"| rag

Routing flow

sequenceDiagram
    participant CE as CloudEvent
    participant Orch as Orchestrator
    participant Reg as AgentRegistry
    participant Spec as Specialist
    participant MCP as MCP Tools

    CE->>Orch: de.billing.rechnung.erstellt
    Orch->>Reg: find_specialist(event_type)
    Note over Reg: trigger_patterns match?
    Reg-->>Orch: billing-regulatory-guard-agent
    Orch->>Spec: run(event_data)
    loop ReAct loop (max_turns)
        Spec->>MCP: get_billing_record(record_id)
        MCP-->>Spec: Rechnung JSON
        Spec->>MCP: validate_tariff_config(tariff)
        MCP-->>Spec: compliance warnings
        Note over Spec: Structure output
    end
    Spec-->>Orch: AgentDecision {outcome, summary}
    Orch-->>CE: de.agent.decision.made (audit)

Parallel dispatch flow

sequenceDiagram
    participant CE as CloudEvent
    participant Orch as Orchestrator
    participant A1 as billing-anomaly-agent
    participant A2 as billing-regulatory-guard-agent
    participant MCP as MCP Tools

    CE->>Orch: de.billing.rechnung.erstellt
    Note over Orch: dispatch_mode = parallel
    Note over Orch: 2 specialists match trigger
    par Concurrent execution
        Orch->>A1: run(event_data)
        A1->>MCP: check_billing_anomaly
        MCP-->>A1: deviation_pct: 35%
        A1-->>Orch: ANOMALY_DETECTED: WARNING
    and
        Orch->>A2: run(event_data)
        A2->>MCP: get_billing_record
        MCP-->>A2: Rechnung JSON
        A2->>MCP: validate_tariff_config
        MCP-->>A2: §42 ENERGIEMIX_MISSING
        A2-->>Orch: COMPLIANCE_STATUS: WARNINGS
    end
    Note over Orch: Merge all AgentDecisions
    Orch-->>CE: de.agent.decision.made (merged)

Agent Mesh

agentd uses the Orchestrator + Specialist Mesh pattern:

Orchestrator receives the trigger and either:
- Matches a trigger_pattern glob → routes directly to the specialist
- Asks the LLM to triage → specialist selection via transfer_to_{specialist} tool call
- Answers directly if no specialist applies
Specialist agents run a ReAct loop (Reason → Act → Observe):
- Each iteration calls one or more MCP tools
- Observes tool results and decides next action
- Continues until a Text result or a Handoff to another specialist
Handoffs are followed up to 3 hops. Each hop re-runs the full ReAct loop with the new specialist's system prompt and tool set.

Bundled specialists

All provider/model assignments are operator-configured via [bundled_agents] default_model and [bundled_agents.overrides.<name>]. The built-in definitions contain only the system prompt, default trigger patterns, and default MCP server requirements.

Specialist	Default triggers	Default MCP servers
`mako-agent`	`de.mako.process.`, `de.mako.aperak.`	makod, marktd, processd, obsd
`deadline-alert-agent`	`de.mako.process.failed`, `de.mako.aperak.timeout`, `de.obs.stp.parity.alert`	obsd, makod, marktd
`billing-agent`	`de.invoic.receipt.disputed`, `de.accounting.*`	invoicd, billingd, accountingd, netzbilanzd
`netzbilanz-agent`	`de.netzbilanz.invoic.drafted`, `de.netzbilanz.invoic.dispatched`	netzbilanzd, marktd, edmd
`invoice-reconciliation-agent`	`de.invoic.payment.overdue`, `de.invoic.receipt.disputed`	invoicd, marktd, netzbilanzd
`billing-anomaly-agent`	`de.billing.rechnung.erstellt`	billingd, edmd
`billing-regulatory-guard-agent`	`de.billing.rechnung.erstellt`	billingd, marktd
`jahresabrechnung-agent`	manual trigger	billingd, edmd, marktd
`eeg-agent`	`de.eeg.anlage.foerderung-auslaufend`, `de.messwert.reading.direct.stored`	einsd, edmd, marktd
`eeg-compliance-agent`	`de.eeg.anlage.`, `de.eeg.verguetung.`, `de.eeg.marktpraemie.*`	einsd, obsd
`payment-reconciliation-agent`	`de.accounting.payment.due`, `de.accounting.bankruecklast`	accountingd
`compliance-agent`	`de.obs.stp.parity.alert`	obsd, processd, marktd, invoicd
`msb-history-agent`	`de.messwert.reading.quality.warning`, `de.messwert.reading.direct.stored`	edmd, makod, marktd
`meter-data-agent`	`de.messwert.reading.quality.warning`, `de.mako.process.completed`	edmd, marktd
`grid-anomaly-agent`	`de.markt.nb-contract.updated`, `de.markt.malo.updated`	marktd, obsd
`tariff-optimization-agent`	`de.billing.rechnung.erstellt`, `de.mako.process.completed`	billingd, tarifbd, edmd, marktd
`vertragd-agent`	`de.vertrag.*`, `de.mako.aperak.rejected`, `de.mako.process.failed`, `de.vertrag.ablauf.ankuendigung`, `de.vertrag.preisaenderung.ankuendigung`	vertragd, processd, marktd
`tarifbd-agent`	`de.tarif.product.updated`, `de.tarif.angebot.abgelaufen`, `de.tarif.epex.missing`	tarifbd, marktd
`processd-agent`	`de.mako.process.initiated`, `de.mako.process.rejected`	processd, marktd, obsd
`sperrd-agent`	`de.sperr.*`, `de.mako.process.completed`	sperrd, makod, marktd
`portald-agent`	`de.billing.rechnung.erstellt`, `de.eeg.anlage.foerderung-auslaufend`, `de.accounting.mahnung.issued`	portald, billingd, einsd, accountingd
`regulatory-reporting-agent`	manual / scheduled	obsd, processd, invoicd, marktd
`replacement-value-agent`	`de.messwert.reading.quality.warning`, `de.mako.process.completed`	edmd, marktd, obsd
`mabis-syncd-agent`	`de.messwert.reading.quality.warning`	edmd, obsd, marktd
`smgw-diagnostics-agent`	`de.messwert.cls.compliance-issue`, `de.messwert.reading.quality.warning`, `de.messwert.reading.direct.stored`, `de.mako.process.initiated`, `de.markt.geraet.konfiguration.updated`	edmd, marktd, obsd, processd
`vpp-billing-agent`	`de.vpp.dispatch.confirmed`, `de.vpp.settlement.berechnet`	billingd, marktd, obsd
`gabi-gas-agent`	`de.gabi.imbalance.`, `de.gabi.alocat.missing`, `de.gabi.nomination.`, `de.netzbilanz.invoic.drafted`	makod, netzbilanzd, marktd, obsd
`einsd-batch-agent`	`de.eeg.settlement.batch-due`, `de.eeg.compliance.*`, `de.eeg.anlage.foerderung-auslaufend`	einsd, edmd, tarifbd, obsd

All 28 specialist definitions are compiled into the agentd binary. Activate them via [bundled_agents] in agentd.toml — no system prompt copy-paste required. See the Configuration section below for a full example.

LLM Providers

Provider	Backend string	Notes
OpenAI	`openai`	`text-embedding-3-small` for embeddings; compatible with Azure OpenAI, Ollama, LM Studio
Anthropic	`anthropic`	Claude (operator-configured model); BM25 keyword fallback (no embedding API)
AWS Bedrock	`bedrock`	SigV4 signed requests; Claude on Bedrock or Titan embeddings

RAG (Retrieval-Augmented Generation)

agentd uses LanceDB as its vector store — a Rust-native, serverless vector database that stores embeddings on object storage (S3/GCS/Azure Blob) or locally.

flowchart LR
    SRC["Knowledge sources<br/>(AHB PDFs, runbooks,<br/>BNetzA decisions)"]
    CHUNK["Paragraph-boundary<br/>chunking (512 tokens)"]
    EMBED["Embedding provider<br/>(OpenAI text-embedding-3-small)"]
    LANCE[("LanceDB<br/>S3 / GCS / local<br/>IVF_PQ ANN index")]
    QUERY["Query vector<br/>(question embedding)"]
    RESULT["Top-k chunks<br/>→ system prompt context"]

    SRC --> CHUNK --> EMBED --> LANCE
    QUERY --> LANCE --> RESULT

BM25 fallback: When using Anthropic (no embedding API), agentd runs keyword search over all stored chunks. Suitable for knowledge bases up to ~50,000 chunks.

Storage URI examples:

storage_uri       = "./data/rag"              # local (dev)
storage_uri       = "s3://my-bucket/rag"      # AWS S3
storage_uri       = "gs://my-bucket/rag"      # Google Cloud Storage
storage_uri       = "az://my-container/rag"   # Azure Blob
score_threshold   = 0.3   # min cosine similarity — low-quality chunks filtered out

All RAG queries are tenant-scoped — documents indexed by Tenant A are never returned to Tenant B. The LanceDB schema includes a tenant column with a per-query filter.

Dead-letter queue (DLQ)

Agent sessions with outcome "error" or "timeout" are placed in a bounded in-memory DLQ. A background worker retries them with exponential backoff.

de.billing.rechnung.erstellt
    → agent session fails (DB unavailable)
    → pushed to DLQ
    → retry in 30s, 90s, 270s, 810s (base_backoff_secs × 3^attempt)
    → after 4 failures: EXHAUSTED
    → emits de.agent.session.dlq.exhausted CloudEvent to audit webhook

# Inspect current DLQ
curl "http://agentd:9580/api/v1/dlq"

Response:

{
  "pending_count": 2,
  "exhausted_count": 1,
  "entries": [...],
  "recent_exhausted": [{ "event_type": "...", "attempts": 4, "last_error": "..." }]
}

Configure via [dlq] in agentd.toml:

[dlq]
capacity         = 100   # max DLQ depth (dropped silently beyond this — log WARN)
max_retries      = 4     # attempts before EXHAUSTED
base_backoff_secs = 30   # retry delays: 30s, 90s, 270s, 810s

Security

Concern	Implementation
`POST /api/v1/run` auth	OIDC/JWT via `Claims` extractor; dev mode emits `[WARN]`
Inbound webhook HMAC	`X-Mako-Signature: sha256=...` verified when `inbound_hmac_secret` set; constant-time compare; 403 on mismatch
Max concurrent sessions	`max_sessions` semaphore; 429 when exhausted
Session timeout	`session_timeout_secs` wall-clock cap (default 300 s)
API keys	`api_key`, `mcp_api_key`, `aws_secret_access_key`, `audit_hmac_secret` stored as `SecretString` — never in logs or debug output
RAG tenant isolation	Every LanceDB query filtered to `tenant` column

Configuration

Minimal — enable built-in specialists

# agentd.toml — using built-in specialist catalog
tenant = "9900357000004"

[providers.openai]
backend = "openai"
api_key = "env:OPENAI_API_KEY"

[orchestrator]
provider   = "openai"
model      = "gpt-4o"
max_turns  = 10
dispatch_mode = "sequential"  # sequential | parallel | race

# ── Enable built-in specialists ───────────────────────────────────────────────
[bundled_agents]
enable_all       = true          # activate all 28 built-in specialists
default_provider = "openai"
default_model    = "gpt-4o-mini"

# Upgrade specific agents to more capable models
[bundled_agents.overrides.mako-agent]
model = "gpt-4o"

[bundled_agents.overrides.jahresabrechnung-agent]
model     = "gpt-4o"
max_turns = 20

[mcp_servers]
makod    = "http://makod:8080/mcp"
marktd   = "http://marktd:8180/mcp"
billingd = "http://billingd:9280/mcp"
edmd     = "http://edmd:8380/mcp"
obsd     = "http://obsd:8480/mcp"
# ... every MCP-exposing service (15 — mabis-syncd has no MCP server)
mcp_api_key = "env:AGENTD_MCP_API_KEY"

trigger_event_types = [
  "de.mako.process.failed",
  "de.billing.rechnung.erstellt",
  "de.eeg.*",
  "de.invoic.receipt.disputed",
]

# ── Security ──────────────────────────────────────────────────────────────────
# Inbound webhook HMAC (strongly recommended in production)
inbound_hmac_secret = "env:AGENTD_INBOUND_HMAC_SECRET"

# OIDC (optional — dev mode when absent, POST /api/v1/run accepts all)
[oidc]
issuer   = "https://keycloak:8080/realms/mako"
audience = "agentd"

# Session limits
max_sessions        = 20    # concurrent sessions (429 beyond this)
session_timeout_secs = 300  # 5 minutes per session wall-clock limit

# ── Dead-letter queue ─────────────────────────────────────────────────────────
[dlq]
capacity          = 100
max_retries       = 4
base_backoff_secs = 30

Custom agents — override or extend built-ins

# Custom agent that overrides billing-anomaly-agent with stricter threshold
[[agents]]
name      = "billing-anomaly-agent"
specialty = "Billing anomaly detection — strict mode (10% threshold)"
provider  = "openai"
model     = "gpt-4o"
max_turns = 12
mcp_servers = ["billingd", "edmd"]
trigger_patterns = ["de.billing.rechnung.erstellt"]
system_prompt = """
You are the billing anomaly detection specialist (strict mode: 10% threshold).
# ... your custom prompt ...
"""

Parallel dispatch — compliance events

# Fan out to ALL specialists matching the event type simultaneously
[orchestrator]
dispatch_mode  = "parallel"
parallel_limit = 4  # max concurrent specialists

# billing.rechnung.erstellt triggers BOTH:
# - billing-anomaly-agent    (deviation check)
# - billing-regulatory-guard-agent  (§40/§41/§41a compliance)
# simultaneously, returning aggregated results

Triggering an agent run

Via CloudEvent webhook:

curl -X POST http://agentd:9580/webhook \
  -H "Content-Type: application/cloudevents+json" \
  -d '{
    "specversion": "1.0",
    "type": "de.billing.rechnung.disputed",
    "source": "urn:mako:billingd:tenant:9900357000004",
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "input": { "malo_id": "51238696780", "record_id": "...", "reason": "check 4 failed" }
  }'

Manual run:

curl -X POST http://agentd:9580/api/v1/run \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "manual.billing.dispute-analysis",
    "input": { "malo_id": "51238696780", "context": "Invoice R2026-001 disputed" }
  }'

CloudEvents emitted

Event type	When
`de.agent.decision.made`	Agent completes a run (includes decision text + tools used)

Audit webhook

Every session's de.agent.decision.made CloudEvent is pushed to the ring buffer and, when configured, POSTed to an external sink:

audit_webhook_url = "https://erp.example/hooks/agent-decisions"
audit_hmac_secret = "env:AGENTD_AUDIT_HMAC"  # X-Mako-Signature (HMAC-SHA256)

Endpoints

Method	Path	Description
`POST`	`/webhook`	Inbound CloudEvent trigger
`POST`	`/api/v1/run`	Manual agent invocation
`GET`	`/api/v1/sessions`	Last 100 agent decisions (in-memory ring buffer)
`GET`	`/api/v1/agents`	List all active agents (built-in + custom) with capabilities
`GET`	`/api/v1/agents/catalog`	Full catalog of all 28 built-in definitions (even if not enabled)
`GET`	`/.well-known/agents/{name}`	A2A Agent Card for a named specialist
`POST`	`/api/v1/rag/ingest`	Index a live text document into LanceDB
`POST`	`/api/v1/rag/search`	Query the RAG knowledge base directly
`GET`	`/health`	Liveness
`GET`	`/health/ready`	Readiness

Edit this page ↗