Skip to main content
Alert rules monitor platform metrics against thresholds across a rolling time window. When a rule triggers, the platform fans out a notification to every destination the rule references.
Automatic firing requires the cron to be wired. Alert rules are evaluated only when POST /api/cron/tick is invoked. During pilot / initial deployment the cron is not wired to AWS EventBridge by default, so rules are defined but dormant — they won’t fire on their own.To run an evaluation on demand (for development, manual triggering, or smoke-testing a newly created rule), hit /api/cron/tick yourself — see Triggering evaluation manually below. To enable automatic every-minute evaluation, follow EventBridge Setup when you’re ready.
Every rule ships through the same pipeline:
  1. A caller (EventBridge, a manual curl, or any other scheduler) hits /api/cron/tick with the shared CRON_TOKEN.
  2. The tick iterates tenants and calls evaluateAllAlertRules per tenant.
  3. The evaluator computes the observed metric value over windowMinutes, compares against threshold using operator, and — if triggered — fans out a notification to every destination listed in channels (as long as the rule is outside its cooldownMinutes).
  4. lastFiredAt is updated and an audit log entry is written.

​
Supported metrics

Each observation is computed against two windows: the current window (ending now) and the baseline window (same width, immediately preceding). The evaluator uses the baseline to derive severity — a bigger breach yields a higher severity tier.

​
Event-injected metrics

The metrics above are computed over a window by the cron tick. A second, smaller class of metrics is injected at the moment an event happens rather than polled — the platform pushes a value straight to the evaluator, which fires any enabled rule whose metric matches. Wire a rule against one of these to be notified the instant it occurs. A rule that pages the moment the decision plane starts failing open:
On a Redis outage the decision routes deliberately allow the request (availability over strict limiting) — a genuine over-limit with Redis healthy still returns 429 and never raises rate_limit_degraded. See Recommend API → rate limiting.

​
Operators

​
Severity

When a rule fires, the evaluator derives severity from the ratio |observed − threshold| / |threshold| (falling back to baseline when threshold = 0):
  • ≥ 1.0 → critical
  • ≥ 0.5 → warning
  • else → info
Severity is passed to the notification payload so adapters render the right visual treatment (e.g., themeColor in Teams, severity emoji in Slack, colored band in ops email).

​
Cooldown

Every rule has a cooldownMinutes knob. After a rule fires, subsequent evaluations that would otherwise trigger are recorded with status = "cooldown" and no notification is dispatched until lastFiredAt + cooldownMinutes has passed. This prevents paging storms when a metric bounces across the threshold.

​
Default rules for new tenants

Every newly registered tenant is seeded with two enabled alert rules, both targeting the registering admin’s email address. Without them a fresh tenant has no rules, so 5xx spikes and scoring degradation would be invisible until someone configured alerting by hand. Edit or disable them in Settings → Alert Rules.

​
Email destinations

An email-type destination ({ "type": "email", "target": "ops@example.com" }) delivers through the platform’s SES sender — the same sender used for auth emails. Configure SES_FROM_EMAIL and AWS credentials for delivery to succeed; see Environment Variables.

​
Configure a rule

Open Settings → Alert Rules and click New Rule. Fields:
  • Name — free text; included in notification titles.
  • Metric — one of the supported metrics above.
  • Operator / Threshold — comparison to evaluate.
  • Window (minutes) — observation window.
  • Cooldown (minutes) — minimum gap between consecutive fires.
  • Destinations — multi-select of Notification Destinations; every selected destination receives the alert on fire.
  • Enabled — toggle to pause evaluation without deleting the rule.
Rules must reference at least one destination. If you delete a destination, rules pointing at it will log delivery_failed on their next fire.
Creating a rule in the UI does not cause it to start firing on its own. Until /api/cron/tick is invoked (manually or via EventBridge), the rule sits idle. See Triggering evaluation manually and EventBridge Setup.

​
Example rule payloads

A rule that pages when p99 decision latency crosses 500ms over a 10-minute window:
A rule that alerts when acceptance rate drops below 5% over 5 minutes:
Each string in channels is the UUID of a configured notification provider. The legacy {type, target} shape remains supported for backward compatibility; prefer provider IDs for new rules.

​
Rule lifecycle

  • status = "ok" — last evaluation did not trigger.
  • status = "fired" — last evaluation triggered and at least one destination accepted the dispatch.
  • status = "cooldown" — last evaluation triggered but the rule is still within cooldown.
  • status = "delivery_failed" — last evaluation triggered but every destination returned a failure.
  • status = "unsupported_metric" — metric name is not recognized (the rule never fires until fixed).
A rule that is never evaluated stays at whatever status value it last had (or the default). Dormant rules don’t transition states on their own — only a tick evaluation can move them.

​
Triggering evaluation manually

When the cron is not wired to EventBridge (e.g., during pilot, local development, or to smoke-test a newly created rule), you can invoke the evaluator directly:
This runs a single evaluation pass across every enabled rule for every tenant. Each invocation is independent — rules still respect cooldownMinutes, so two calls back-to-back will not double-fire. Response:

​
Wire automatic evaluation

When you’re ready for rules to evaluate on a cadence without manual invocation, follow EventBridge Setup. That page is marked optional on purpose — automatic firing is a pilot graduation step, not a prerequisite.