> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Scaling & Performance
> How to scale KaireonAI for production workloads
## Request Lifecycle
Every Recommend API request passes through a series of stages. Understanding where
caching and rate limiting apply helps you tune for your workload.
```mermaid theme={null}
sequenceDiagram
participant Client
participant API as API Server
participant RL as Rate Limiter (Redis)
participant DB as PostgreSQL
participant Cache as Redis Cache
participant Model as Scoring Model
Client->>API: POST /api/v1/recommend
API->>RL: Check rate limit (sorted-set sliding window)
RL-->>API: allowed / 429
API->>API: Tenant auth (X-Tenant-Id + X-API-Key)
API->>Cache: Resolve flow route + load flow config
Cache-->>API: Cached flow (or DB fetch, 5-min TTL)
API->>Cache: Load active offers, decisioning gates, contact policies
Cache-->>API: Cached entities (5-min TTL)
rect rgb(40, 40, 60)
Note over API,Model: Pipeline Execution
API->>Cache: Enrichment lookup (configurable TTL per source)
Cache-->>API: Customer data (or DB query + cache)
API->>API: Qualification filtering
API->>API: Contact policy enforcement
API->>Model: Score candidates
Model-->>API: Scores (or degraded fallback on failure)
API->>API: Ranking, guardrails, budget pacing
end
API-->>Client: JSON response
```
## Caching Strategy
KaireonAI uses Redis as its caching layer. All cache reads go through a single
read-through helper that transparently handles cache misses by querying
PostgreSQL and storing the result with a configurable TTL, so callers never
need to manage cache invalidation themselves.
### What gets cached
| Data | Cache Key Pattern | Default TTL | Notes |
| ------------------- | ------------------------------------------- | ------------------------------------- | ------------------------------------------ |
| Active offers | `t:{tenantId}:offers:active` | 300s (5 min) | Includes creatives, categories, placements |
| Decisioning gates | `t:{tenantId}:policies:eligibility` | 300s (5 min) | Active rules ordered by priority |
| Contact policies | `t:{tenantId}:policies:contactPolicy` | 300s (5 min) | Active policies ordered by priority |
| Guardrail rules | `t:{tenantId}:guardrails:active` | 30s | Shorter TTL for faster policy iteration |
| Enrichment data | `enrich:{tenantId}:{customerId}:{schemaId}` | Configurable per source (default 60s) | Per-customer, per-schema-source |
| Rate limit counters | `ratelimit:{route}:{tenantId}:{identifier}` | Window duration + 1s | Redis sorted sets |
| Cap counters | `kaireon:cap:{key}` | End of current UTC day | Auto-expire at midnight UTC |
### Tuning TTLs
Enrichment sources support per-source TTL configuration in the Decision Flow's
enrichment stage:
```json theme={null}
{
"enrichment": {
"sources": [
{
"schemaId": "customer_profile",
"fields": ["loan_amount", "credit_score"],
"cacheTtlSeconds": 300,
"prefix": "customer"
},
{
"schemaId": "real_time_signals",
"fields": ["last_page_viewed"],
"cacheTtlSeconds": 10,
"prefix": "signals"
}
]
}
}
```
Set `cacheTtlSeconds` lower for volatile data (real-time signals, session context)
and higher for stable data (customer demographics, account details).
### Cache Invalidation
Entity caches (offers, rules, policies) use a TTL-based expiration strategy.
After updating an offer or policy via the CRUD API, changes propagate within
the TTL window (up to 5 minutes for offers/policies, 30 seconds for guardrails).
For immediate invalidation, restart the API process or reduce the TTL via
environment configuration.
## Rate Limiting
The Recommend API enforces per-tenant, per-endpoint rate limiting using a
**Redis sorted-set sliding window** algorithm.
### Algorithm
1. **ZREMRANGEBYSCORE** -- remove entries outside the current window
2. **ZADD** -- add the current request with its timestamp as score
3. **ZCARD** -- count entries remaining in the window
4. **EXPIRE** -- set key TTL to window duration + 1 second for cleanup
All four operations execute in a single Redis pipeline for atomicity. A 500ms
timeout protects against Redis latency -- if Redis does not respond in time,
the limiter falls back to an in-memory sliding window.
### Tier Configuration
Rate limits are tenant-scoped with three built-in tiers:
| Tier | Requests per Minute |
| ------------ | ------------------- |
| `free` | 100 |
| `standard` | 1,000 |
| `enterprise` | 10,000 |
Configure per-tenant tiers via environment variables:
```bash theme={null}
# Global default tier
RATE_LIMIT_TIER=standard
# Per-tenant override (tenant ID uppercased, hyphens to underscores)
RATE_LIMIT_TIER_ACME_CORP=enterprise
```
### Response Headers
When a request is rate-limited, the API returns HTTP 429 with:
| Header | Description |
| ----------------------- | -------------------------------------- |
| `X-RateLimit-Limit` | Maximum requests allowed in the window |
| `X-RateLimit-Remaining` | Requests remaining (0 when limited) |
| `Retry-After` | Seconds until the client should retry |
### Fail-Open vs Fail-Closed
The rate limiter supports two failure modes when Redis is unavailable:
* **Fail-open (default):** Falls back to in-memory rate limiting. Use for
standard API endpoints where availability matters more than strict enforcement.
* **Fail-closed:** Returns 429 when Redis is down. The Recommend API uses
`failOpen: false` to prevent abuse when rate limit state is unavailable.
### Edge-Layer Rate Limiting
For DDoS protection, layer edge-level rate limiting in front of the API:
nginx `limit_req_zone`, AWS WAF rate rules, or Cloudflare rate limiting rules.
The application-level limiter handles tenant-scoped business logic limits;
the edge layer handles volumetric protection.
## Scoring Failure Fallback
The scoring stage does not wrap models in a stateful circuit breaker. A model
error is caught **per request** and a fallback score is applied so the decision
still completes:
1. For an external scoring endpoint that fails, the candidate score becomes
`0.5 * fitMultiplier` (the `0.5` default is a constant in the scoring stage).
2. When a model is missing, scoring falls through to `priority_weighted`
(`priority / 100`).
3. The response carries `degradedScoring: true` (and each affected candidate is
marked) so callers can tell the decision ran degraded.
There is no per-model failure counter, cooldown, or open/closed circuit state in
the scoring path. A separate general-purpose circuit breaker
(`lib/circuit-breaker.ts`, defaults **5 failures / 5-minute cooldown / 3
half-open probes**, state optionally persisted to Redis under `kaireon:cb:`)
guards *outbound integrations* — alert and trigger webhooks, connector tests,
CMS sync, and audit forwarding — not scoring. See
[Operations — Circuit Breakers](/self-host/architecture/operations#circuit-breakers).
### Monitoring
The `kaireon_decision_flow_execution_latency_ms` histogram tracks end-to-end
pipeline latency; `kaireon_scoring_latency_ms` isolates the scoring stage.
Decisions that fell back on a scoring error surface via the `degradedScoring`
flag on the response payload.
## Atomic Cap Checking
The engine ships an `atomicCapCheck` helper (`lib/decision-flow-engine.ts`) that
performs race-free daily counting via **Redis INCR**:
1. `INCR kaireon:cap:{key}` -- single atomic read-and-increment (the caller
supplies `key`, e.g. `mandatory:{offerId}:{YYYY-MM-DD}`)
2. On first increment (`current === 1`), set EXPIRE to end of current UTC day
3. If `current > cap`, the check returns `allowed: false` (the caller supplies
the `cap` value; there is no built-in default)
When Redis is unavailable, it falls back to a caller-provided Prisma count
function. This fallback has a small race window under concurrent requests but is
acceptable for resilience.
This helper is present in the codebase but is **not currently wired into the
active decision pipeline** — the composable pipeline enforces mandatory-offer
and frequency rules through the contact-policy stage instead. Treat this
section as reference for the helper's behavior, not a description of a live
per-decision cap.
## Connection Pooling
KaireonAI uses Prisma 7 with the `@prisma/adapter-pg` driver adapter. Connection
pooling is handled by the underlying `pg` Pool.
### Configuration
The database connection is configured in `prisma.config.ts` via the `DATABASE_URL`
environment variable. Pool sizing is controlled through connection string parameters:
```bash theme={null}
# Example with pool sizing parameters
DATABASE_URL="postgresql://user:pass@host:5432/kaireon?connection_limit=20&pool_timeout=10"
```
### Sizing Guidance
| Deployment Size | Suggested Pool Size | Notes |
| --------------- | ------------------- | ---------------------------------------- |
| Single instance | 5-10 | Default pg Pool settings are sufficient |
| 2-5 replicas | 10-15 per replica | Total connections = replicas x pool size |
| 10+ replicas | 5-10 per replica | Use PgBouncer or RDS Proxy to multiplex |
**Rule of thumb:** Total connections across all replicas should not exceed your
database's `max_connections` minus a buffer for admin/monitoring connections.
## Horizontal Scaling
The KaireonAI API is **stateless** -- all shared state lives in Redis and
PostgreSQL. This means you can scale API instances horizontally with no
coordination overhead.
### Architecture
```
┌─────────────┐
│ Load │
│ Balancer │
└──────┬──────┘
│
┌────────────┼────────────┐
│ │ │
┌─────┴─────┐ ┌───┴─────┐ ┌───┴─────┐
│ API Pod │ │ API Pod │ │ API Pod │
│ (Node.js) │ │ │ │ │
└─────┬──────┘ └───┬─────┘ └───┬─────┘
│ │ │
└────────────┼────────────┘
│
┌────────────┼────────────┐
│ │
┌─────┴─────┐ ┌──────┴──────┐
│ Redis │ │ PostgreSQL │
│ (shared) │ │ (shared) │
└───────────┘ └─────────────┘
```
### Key Properties
* **No session affinity required:** Any API pod can handle any request.
Rate limit state and caching are in Redis; all persistent state is in PostgreSQL.
* **In-memory circuit breakers are per-process:** Each pod tracks its own model
failure counts. This is intentional -- a model failure on one pod does not
cascade to others, and each pod independently probes recovery.
* **In-memory rate limit fallback is per-process:** When Redis is down, each pod
maintains its own rate limit counters. Effective limits become
`configured_limit x num_pods` during Redis outages.
* **Scale API pods independently from worker pods:** Decision API pods handle
synchronous request/response. Data pipeline worker pods handle asynchronous
ETL. Size each tier based on its workload.
* **In-process schedulers need attention at multi-replica:** by default each API
container also runs the internal Flow scheduler and the maintenance (cron)
scheduler in-process (see [Process Model](/self-host/architecture/overview#process-model)).
The Flow scheduler is multi-replica safe via a PostgreSQL advisory lock, so it
is fine to leave enabled on every replica. The maintenance scheduler assumes a
single replica — its jobs are idempotent so duplicate passes are harmless, but
for a clean multi-replica setup disable it (`MAINTENANCE_SCHEDULER_ENABLED=false`)
and drive the `/api/**` cron routes from one external scheduler (Kubernetes
CronJobs or EventBridge). Move queue and outbox processing to dedicated pods
with `WORKER_INPROCESS=0`.
## Batch vs Streaming Pipelines
Data pipelines support two execution modes, configured per-pipeline in the
`executionConfig` JSON field:
### Batch Mode
For scheduled or on-demand data loads:
```json theme={null}
{
"mode": "batch",
"batchSize": 1000,
"parallelism": 4,
"partitioning": {
"strategy": "hash",
"key": "customer_id",
"partitions": 8
}
}
```
| Parameter | Description | Default |
| ------------------------- | -------------------------------------------------- | ------- |
| `batchSize` | Records per processing chunk | 1000 |
| `parallelism` | Concurrent processing threads | 1 |
| `partitioning.strategy` | How to split data (`hash`, `range`, `round_robin`) | None |
| `partitioning.key` | Field to partition on | -- |
| `partitioning.partitions` | Number of partitions | -- |
### Streaming Mode (planned)
**Streaming mode is a placeholder and not yet implemented.** The execution
mode field accepts `streaming` for forward-compatibility, but the platform
does not currently spawn a long-lived consumer process. Kafka, Confluent,
and (when shipped) Amazon Kinesis connectors run as **batch polling** —
each pipeline run opens a consumer, reads up to `maxMessages` records,
commits offsets, and closes. Schedule those pipelines on a cron cadence
that matches your freshness target until a persistent worker is available.
The config shape below is reserved for the future streaming runtime — today
it has no effect beyond being persisted on the pipeline record:
```json theme={null}
{
"mode": "streaming",
"batchSize": 100,
"parallelism": 2,
"checkpointIntervalMs": 30000
}
```
### K8s Worker Pod Configuration
Pipeline execution runs on dedicated worker pods, separate from the API tier.
Configure resource limits based on pipeline complexity:
* **CPU-bound transforms** (expression evaluation, hashing): Scale `parallelism`
up to available CPU cores.
* **I/O-bound transforms** (external lookups, PII masking): Higher `parallelism`
with moderate CPU allocation.
* **Memory-bound transforms** (large batch joins): Increase pod memory limits
and reduce `batchSize`.
## Production Tuning Checklist
### Under 1K decisions/day
* Single API instance with default settings
* In-memory rate limiting and caching are sufficient
* Default connection pool (5-10 connections)
* No Redis required (in-memory fallbacks handle the load)
### 1K -- 100K decisions/day
* **Redis required** for rate limiting, enrichment caching, and atomic cap checks
* Tune enrichment TTLs: increase to 300s+ for stable customer data
* Increase connection pool to 15-20 per instance
* Enable decision tracing with a sample rate (e.g., 10%) rather than 100%
* Monitor `decisionLatencyMs` and `scoringModelFailureTotal` metrics
### 100K+ decisions/day
* **Multiple API replicas** behind a load balancer (3+ pods recommended)
* **Dedicated Redis instance** (ElastiCache or equivalent) with sufficient memory
for rate limit sorted sets + enrichment cache
* **PostgreSQL read replicas** for offer/policy reads; primary for writes only
* Use **PgBouncer or RDS Proxy** to multiplex database connections
* Keep the offer scan set bounded by tightening the `inventory` node's scope and status filters (it has no built-in scan limit)
* Lower guardrail TTL if policy changes need sub-30s propagation
* Configure edge-layer rate limiting (AWS WAF, Cloudflare) for DDoS protection
* Enable budget pacing for high-volume offers to spread delivery across the day
* Override the model-degraded fallback score in the decision flow engine if your
scoring distribution warrants a different default than 0.5
## Environment Variables Reference
| Variable | Description | Default |
| ----------------- | ---------------------------- | ------------------------ |
| `REDIS_URL` | Redis connection string | `redis://localhost:6379` |
| `DATABASE_URL` | PostgreSQL connection string | -- (required) |
| `RATE_LIMIT_TIER` | Global rate limit tier | `standard` |
The decisioning defaults below are currently compiled-in constants, not
environment variables. To change one, edit the corresponding constant in the
circuit-breaker module or the decision-flow engine module and redeploy:
| Default | Value | Where it lives |
| ------------------------------- | ----- | ------------------------------------------------- |
| Score when model is unavailable | 0.5 | Scoring stage constant (`lib/pipeline-runner.ts`) |
## Related
System architecture and module layout
Monitoring, dashboards, and alerting
How the decision pipeline works