> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Alert Rules

> Define metric thresholds that fire notifications to configured destinations when breached, with cooldown support.

Alert rules monitor platform metrics against thresholds across a rolling
time window. When a rule triggers, the platform fans out a notification to
every destination the rule references.

<Note>
  **Automatic firing requires the cron to be wired.** Alert rules are evaluated
  only when `POST /api/cron/tick` is invoked. During pilot / initial deployment
  the cron is **not** wired to AWS EventBridge by default, so rules are defined
  but dormant — they won't fire on their own.

  To run an evaluation on demand (for development, manual triggering, or
  smoke-testing a newly created rule), hit `/api/cron/tick` yourself — see
  [Triggering evaluation manually](#triggering-evaluation-manually) below. To
  enable automatic every-minute evaluation, follow
  [EventBridge Setup](../self-host/deploy/eventbridge-setup) when you're ready.
</Note>

Every rule ships through the same pipeline:

1. A caller (EventBridge, a manual `curl`, or any other scheduler) hits `/api/cron/tick` with the shared `CRON_TOKEN`.
2. The tick iterates tenants and calls `evaluateAllAlertRules` per tenant.
3. The evaluator computes the observed metric value over `windowMinutes`, compares against `threshold` using `operator`, and — if triggered — fans out a notification to every destination listed in `channels` (as long as the rule is outside its `cooldownMinutes`).
4. `lastFiredAt` is updated and an audit log entry is written.

## Supported metrics

| Metric                  | Unit           | Window semantics                                                                                                                          |
| ----------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `acceptance_rate`       | rate (0–1)     | positives / impressions over the window                                                                                                   |
| `ctr`                   | rate (0–1)     | clicks (click + convert) / impressions                                                                                                    |
| `revenue`               | currency units | sum of `outcome.conversionValue`                                                                                                          |
| `selection_frequency`   | rate (0–1)     | decisions with at least one selected offer / total decision traces                                                                        |
| `latency_p99`           | milliseconds   | 99th percentile of `DecisionTrace.totalLatencyMs`                                                                                         |
| `degraded_scoring_rate` | rate (0–1)     | traces with `degradedScoring=true` / total traces                                                                                         |
| `http_5xx_count`        | count          | process-wide number of `5xx` API responses in the window (recorded by the metrics wrapper on every route)                                 |
| `suppression_rate`      | rate (0–1)     | qualified candidates removed by contact policy (`afterQualification − afterContactPolicy`) / qualified candidates, summed over the window |
| `empty_candidate_rate`  | rate (0–1)     | decision traces returning zero offers (`finalCount = 0`) / total decision traces                                                          |

<Tip>
  Each observation is computed against **two** windows: the current window
  (ending now) and the baseline window (same width, immediately preceding).
  The evaluator uses the baseline to derive severity — a bigger breach yields
  a higher severity tier.
</Tip>

### Event-injected metrics

The metrics above are computed over a window by the cron tick. A second,
smaller class of metrics is **injected at the moment an event happens**
rather than polled — the platform pushes a value straight to the evaluator,
which fires any enabled rule whose `metric` matches. Wire a rule against one
of these to be notified the instant it occurs.

| Metric                | Injected value | Fires when                                                                                                                                                                                                                                                                    |
| --------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rate_limit_degraded` | `1`            | The rate limiter cannot reach Redis, so `/recommend` and `/respond` **fail open** — decisions keep flowing rather than returning `429`, and this signal is raised so the outage is visible. Throttled to at most once per tenant per 60s (on top of the rule's own cooldown). |

A rule that pages the moment the decision plane starts failing open:

```json theme={null}
{
  "name": "Rate limiter degraded (Redis down)",
  "metric": "rate_limit_degraded",
  "operator": "gte",
  "threshold": 1,
  "windowMinutes": 5,
  "cooldownMinutes": 15,
  "channels": ["11111111-2222-3333-4444-555555555555"],
  "enabled": true
}
```

<Note>
  On a Redis outage the decision routes deliberately **allow** the request
  (availability over strict limiting) — a genuine over-limit with Redis
  healthy still returns `429` and never raises `rate_limit_degraded`. See
  [Recommend API → rate limiting](/api-reference/recommend).
</Note>

## Operators

| Operator | Meaning               |
| -------- | --------------------- |
| `gt`     | observed > threshold  |
| `lt`     | observed \< threshold |
| `gte`    | observed ≥ threshold  |
| `lte`    | observed ≤ threshold  |
| `eq`     | observed = threshold  |

## Severity

When a rule fires, the evaluator derives severity from the ratio
`|observed − threshold| / |threshold|` (falling back to baseline when
`threshold = 0`):

* `≥ 1.0` → **critical**
* `≥ 0.5` → **warning**
* else → **info**

Severity is passed to the notification payload so adapters render the
right visual treatment (e.g., `themeColor` in Teams, severity emoji in
Slack, colored band in ops email).

## Cooldown

Every rule has a `cooldownMinutes` knob. After a rule fires, subsequent
evaluations that would otherwise trigger are recorded with
`status = "cooldown"` and no notification is dispatched until
`lastFiredAt + cooldownMinutes` has passed.

This prevents paging storms when a metric bounces across the threshold.

## Default rules for new tenants

Every newly registered tenant is seeded with two enabled alert rules, both
targeting the registering admin's email address. Without them a fresh tenant
has no rules, so 5xx spikes and scoring degradation would be invisible until
someone configured alerting by hand. Edit or disable them in
**Settings → Alert Rules**.

| Name                            | Metric                  | Operator | Threshold | Window | Cooldown |
| ------------------------------- | ----------------------- | -------- | --------- | ------ | -------- |
| HTTP 5xx spike                  | `http_5xx_count`        | `gte`    | `25`      | 5 min  | 30 min   |
| Degraded scoring rate above 25% | `degraded_scoring_rate` | `gte`    | `0.25`    | 15 min | 60 min   |

## Email destinations

An `email`-type destination (`{ "type": "email", "target": "ops@example.com" }`)
delivers through the platform's SES sender — the same sender used for auth
emails. Configure `SES_FROM_EMAIL` and AWS credentials for delivery to
succeed; see [Environment Variables](/self-host/configure/env-vars).

## Configure a rule

Open **Settings → Alert Rules** and click **New Rule**. Fields:

* **Name** — free text; included in notification titles.
* **Metric** — one of the supported metrics above.
* **Operator / Threshold** — comparison to evaluate.
* **Window (minutes)** — observation window.
* **Cooldown (minutes)** — minimum gap between consecutive fires.
* **Destinations** — multi-select of [Notification Destinations](./notifications); every selected destination receives the alert on fire.
* **Enabled** — toggle to pause evaluation without deleting the rule.

<Warning>
  Rules must reference at least one destination. If you delete a destination,
  rules pointing at it will log `delivery_failed` on their next fire.
</Warning>

<Warning>
  Creating a rule in the UI does **not** cause it to start firing on its own.
  Until `/api/cron/tick` is invoked (manually or via EventBridge), the rule
  sits idle. See [Triggering evaluation manually](#triggering-evaluation-manually)
  and [EventBridge Setup](../self-host/deploy/eventbridge-setup).
</Warning>

## Example rule payloads

A rule that pages when p99 decision latency crosses 500ms over a 10-minute window:

```json theme={null}
{
  "name": "p99 latency spike",
  "metric": "latency_p99",
  "operator": "gt",
  "threshold": 500,
  "windowMinutes": 10,
  "cooldownMinutes": 30,
  "channels": ["11111111-2222-3333-4444-555555555555"],
  "enabled": true
}
```

A rule that alerts when acceptance rate drops below 5% over 5 minutes:

```json theme={null}
{
  "name": "Acceptance rate drop",
  "metric": "acceptance_rate",
  "operator": "lt",
  "threshold": 0.05,
  "windowMinutes": 5,
  "cooldownMinutes": 60,
  "channels": [
    "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
    { "type": "webhook", "target": "https://legacy.example.com/hook" }
  ]
}
```

Each string in `channels` is the UUID of a configured notification
provider. The legacy `{type, target}` shape remains supported for
backward compatibility; prefer provider IDs for new rules.

## Rule lifecycle

* `status = "ok"` — last evaluation did not trigger.
* `status = "fired"` — last evaluation triggered and at least one destination accepted the dispatch.
* `status = "cooldown"` — last evaluation triggered but the rule is still within cooldown.
* `status = "delivery_failed"` — last evaluation triggered but every destination returned a failure.
* `status = "unsupported_metric"` — metric name is not recognized (the rule never fires until fixed).

<Note>
  A rule that is never evaluated stays at whatever `status` value it last had
  (or the default). Dormant rules don't transition states on their own — only
  a tick evaluation can move them.
</Note>

## Triggering evaluation manually

When the cron is not wired to EventBridge (e.g., during pilot, local
development, or to smoke-test a newly created rule), you can invoke the
evaluator directly:

```bash theme={null}
curl -X POST https://your-deployment/api/cron/tick \
  -H "x-cron-token: $CRON_TOKEN"
```

This runs a single evaluation pass across every enabled rule for every
tenant. Each invocation is independent — rules still respect
`cooldownMinutes`, so two calls back-to-back will not double-fire.

Response:

```json theme={null}
{
  "ok": true,
  "tenantsProcessed": 3,
  "rulesEvaluated": 12,
  "rulesFired": 2,
  "errors": [],
  "durationMs": 187,
  "timestamp": "2026-04-17T12:34:56.789Z"
}
```

## Wire automatic evaluation

When you're ready for rules to evaluate on a cadence without manual
invocation, follow [EventBridge Setup](../self-host/deploy/eventbridge-setup).
That page is marked optional on purpose — automatic firing is a pilot
graduation step, not a prerequisite.