> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Outbox publisher (Helm)
> W8.3 — dedicated worker pod that polls outbox_events and publishes via EventPublisher. Splits outbox publish latency from the BullMQ worker tier.
## Why a separate tier
The outbox table guarantees at-least-once event delivery: events written
inside a transaction (e.g., `interaction.recorded.v1`, `outcome.recorded`)
are durable even when the configured EventPublisher backend is down or
slow. Without a dedicated publisher tier, the BullMQ-running worker
pods own this loop alongside long-running batch jobs, and a backed-up
batch can starve the publish loop. Splitting these tiers keeps the
publish tail latency independent of batch contention.
## What changed in the respond hot path
The `/api/v1/respond` endpoint no longer publishes events synchronously.
The `interaction.recorded.v1` event is now enqueued inside the same
database transaction that writes the **interaction history** row.
**Behavior-change note**: an outbox row insertion failure now rolls back
the interaction row. This is a **correctness improvement** vs the prior
fail-open path — the system no longer claims outcomes whose downstream
events it can't persist. The cost is that pathological insert failures
(JSON-too-large, constraint violation, mid-tx connection drop) surface
as 500s to the caller instead of silent drops. Operators investigating
"respond returned 500" should check `outbox_events` insert errors first.
## Operator visibility — what surfaces when things go wrong
| Failure mode | Signal | Operator action |
| ------------------------------ | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Publisher pod down | Pod restart count + Pending events backlog (see metric below) | `kubectl describe pod kaireon-outbox-publisher-*` |
| Loop hangs | Liveness probe fails → k8s restarts | `kubectl logs --previous` |
| EventPublisher backend down | Transport `publish failed` ERROR logs, then rising `retryCount` and eventually `dead_letter_events` rows | Check Kafka/Redpanda/Redis health, then replay the DLQ |
| Bad env config | Process exits with code 2 → CrashLoopBackoff | Fix the **OUTBOX\_POLL\_INTERVAL\_MS** env var etc. |
| Stuck `processing` rows | `outbox_events.status='processing' AND now() - updatedAt > OUTBOX_REAPER_STALENESS_SECONDS` | Auto-handled by `outboxReaper` cron — see below |
| Events pending but not moving | `outbox_events.status='pending' AND "nextAttemptAt" > now()` | Expected — they are inside a backoff window. See [Retry schedule](#retry-schedule-and-dead-lettering) |
| Sustained non-transient errors | Process exits with code 3 after 30 consecutive failures | Investigate root cause; pod will CrashLoopBackoff |
### Recommended Prometheus alert
```yaml theme={null}
- alert: OutboxBacklog
expr: |
pg_stat_activity_count{state="pending"} > 0
OR (kaireon_outbox_pending_count > 100)
for: 5m
labels:
severity: warning
annotations:
summary: "Outbox backlog growing — publisher tier may be unhealthy"
```
(The `kaireon_outbox_pending_count` gauge is registered with the platform metrics registry and refreshed by the outbox processor on every poll tick. See [Metrics Reference](/self-host/operate/metrics-reference#kaireon_outbox_pending_count) for the full PromQL alert family.)
## Who drains the outbox
Exactly one of these must be running, or `outbox_events` fills up silently and
nothing is ever published:
| Deployment | Drainer | Configuration |
| -------------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Kubernetes (Helm) | the dedicated `outbox-publisher` pod | `outboxPublisher.enabled: true`; the chart then sets `OUTBOX_POLLER_INPROCESS=0` on the API pods so they do not also poll |
| Single container (App Runner, Docker, hobby) | the API container's in-process poller | on by default — nothing to configure |
`OUTBOX_POLLER_INPROCESS` is **independent of `WORKER_INPROCESS`**. Setting
`WORKER_INPROCESS=0` means "a dedicated worker container consumes the job
queue"; it says nothing about the outbox. Before 2026-08-15 the two were tied
together, so a single-container deployment that opted out of in-process BullMQ
workers also silently lost its outbox drainer.
Symptom of having no drainer: `outbox_events` rows sitting at
`status='pending'` with `retryCount = 0` — never attempted, as opposed to
attempted-and-failing. Check with:
```sql theme={null}
SELECT status, count(*), max("retryCount") FROM outbox_events GROUP BY status;
```
A healthy deployment shows mostly `published`. All-`pending` with
`max = 0` means nothing is polling.
## Retry schedule and dead-lettering
A failed publish is retried on an exponential backoff, and the due-time for the
next attempt is persisted on the row itself in `outbox_events.nextAttemptAt`.
`NULL` means "due now" — the state of every freshly written event and of every
event replayed out of the DLQ.
The batch claim only picks up rows that are actually due:
```sql theme={null}
WHERE status = 'pending'
AND ("nextAttemptAt" IS NULL OR "nextAttemptAt" <= NOW())
```
Backoff is `min(1000ms x 2^(n-1), 60000ms)` for the *n*-th failed attempt, plus
up to 50% jitter. With the default `maxRetries = 5`:
| Failed attempt | Wait before the next one |
| -------------- | -------------------------------------------------------------------------------------- |
| 1st | 1.0–1.5s |
| 2nd | 2.0–3.0s |
| 3rd | 4.0–6.0s |
| 4th | 8.0–12.0s |
| 5th | none — the event is moved to `dead_letter_events` and its row set to `status='failed'` |
So an event whose backend stays down is dead-lettered after 5 attempts spanning
roughly 15–22 seconds. The DLQ write and the status update happen in one
transaction, and the resulting `ERROR` log carries `dlqDepth` plus an `alert`
level of `INFO` / `WARNING` (>10) / `CRITICAL` (>100).
`nextAttemptAt` is deliberately separate from `updatedAt`. The claim UPDATE
stamps `updatedAt` — that is what the [reaper](#outbox-reaper-cron--apiv1cronoutbox-reaper)
reads to find rows orphaned by a dead worker. Anchoring the backoff on the same
column made the claim overwrite the value it was about to read, so no event with
`retryCount > 0` was ever retried or dead-lettered. Fixed 2026-08-15.
To inspect what a backlog is actually waiting on:
```sql theme={null}
SELECT status, "retryCount", "nextAttemptAt" - NOW() AS due_in, "lastError"
FROM outbox_events
WHERE "tenantId" = '' AND status <> 'published'
ORDER BY "createdAt";
```
## Outbox reaper cron — `/api/v1/cron/outbox-reaper`
A dedicated cron job sweeps `outbox_events` and resets any row stuck in
`processing` whose `updatedAt` is older than the configured staleness
threshold back to `pending`. This closes the failure mode where a worker
dies between claiming a row (UPDATE → `processing`) and either
publishing it or marking it `failed` — without the reaper those rows
would sit in `processing` forever and never re-attempted.
The cron route invokes the outbox processor's stuck-row reaper, which
performs a single bulk SQL UPDATE driven by the configured staleness
threshold. The operation is idempotent — re-running on already-pending
rows is a no-op.
### Helm wiring
```yaml theme={null}
cron:
schedules:
outboxReaper:
enabled: true
schedule: "*/2 * * * *"
path: "/api/v1/cron/outbox-reaper"
```
Wired by default in `helm/values.yaml`. Cadence of 1–5 minutes is fine
because the operation is idempotent.
### Auth
The cron route fail-closes when `CRON_SECRET` is unset (`route.ts:25-32`).
Authenticated callers present the secret via either `Authorization: Bearer ` or `x-cron-secret`. Mismatched values return 401.
### Response
```json theme={null}
{
"status": "ok",
"stalenessSeconds": 300,
"resetCount": 2,
"durationMs": 14,
"timestamp": "2026-04-30T14:20:00.000Z"
}
```
### Configuration
| Variable | Default | Effect |
| --------------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OUTBOX_REAPER_STALENESS_SECONDS` | `300` (5 min) | Rows in `processing` whose `updatedAt` is older than this are reset to `pending`. Invalid or non-positive values fall back to the default with a warning (`route.ts:78-82`). |
| `CRON_SECRET` | unset → 401 | Shared secret for the cron route. Required. |
## Configuration knobs
```yaml theme={null}
outboxPublisher:
enabled: true
replicas: 1
pollIntervalMs: 2000 # tick cadence on idle
shutdownDrainTimeoutMs: 15000 # SIGTERM drain budget
livenessFile: "/tmp/outbox-publisher.alive"
livenessProbe:
enabled: true
initialDelaySeconds: 15
periodSeconds: 30
maxStaleSeconds: 90
failureThreshold: 3
pdb:
enabled: true
minAvailable: 1
```
The publisher pod reads `OUTBOX_LIVENESS_FILE` (default
`/tmp/outbox-publisher.alive`). The liveness probe checks this file's
age — when the publisher loop stops touching it, the probe fails and
Kubernetes restarts the pod.
## Honest known gaps
1. **Structured error IDs shipped on the worker tick path; not yet on every helper.** The publisher's main poll loop mints a per-tick `errorId` and threads it into the next-attempt log line so SIEM tooling can correlate retries. Other in-tier helpers (shutdown drain, reaper companion) still emit bare structured logs and are tracked as a residual for migration. SIEM correlation works for the main loop today.
2. **`kaireon_outbox_pending_count` gauge shipped (W10 wave).** Registered with the platform metrics registry and refreshed every poll tick. The recommended Prometheus alert above can be wired today. `outboxProcessedTotal` + `outboxEventAge` from W8.3 still cover throughput + freshness; this gauge closes the backlog visibility gap. See [Metrics Reference](/self-host/operate/metrics-reference#kaireon_outbox_pending_count) for full alert PromQL.