> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling

> Runbook for scaling KaireonAI — horizontal scaling, database scaling, cache sizing

> **Audience:** SREs, platform operators, DevOps engineers
> **Last updated:** 2026-02-23
> **Infrastructure:** EKS (Kubernetes), RDS PostgreSQL, ElastiCache Redis, PgBouncer

***

## Table of Contents

1. [Scaling the API Layer](#1-scaling-the-api-layer)
2. [Scaling Workers](#2-scaling-workers)
3. [Scaling the Database](#3-scaling-the-database)
4. [Scaling Redis](#4-scaling-redis)
5. [Scaling PgBouncer](#5-scaling-pgbouncer)
6. [Capacity Planning](#6-capacity-planning)

***

## 1. Scaling the API Layer

### Current Architecture

```
ALB (Application Load Balancer)
    |
    v
KaireonAI API Deployment (N replicas, HPA-managed)
    |
    +---> PgBouncer ---> PostgreSQL
    +---> Redis
```

### Horizontal Pod Autoscaler (HPA)

**Default HPA configuration (from Helm chart):**

```yaml theme={null}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: kaireon-api-hpa
  namespace: kaireon
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kaireon-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
```

The default Helm chart provides CPU-based HPA only. For production environments with high throughput, add memory and custom metric scaling via a Helm values override:

<details>
  <summary><strong>Recommended Enhancement: Advanced HPA</strong></summary>

  ```yaml theme={null}
  # values-production-hpa.yaml — apply with: helm upgrade kaireon ./helm/kaireon -f values-production-hpa.yaml
  apiVersion: autoscaling/v2
  kind: HorizontalPodAutoscaler
  metadata:
    name: kaireon-api-hpa
  spec:
    behavior:
      scaleUp:
        stabilizationWindowSeconds: 60
        policies:
          - type: Pods
            value: 4
            periodSeconds: 60
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
          - type: Pods
            value: 2
            periodSeconds: 120
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 60
      - type: Resource
        resource:
          name: memory
          target:
            type: Utilization
            averageUtilization: 75
      - type: Pods
        pods:
          metric:
            name: kaireon_decision_latency_ms
          target:
            type: AverageValue
            averageValue: "150"
  ```

  > **Requires:** Prometheus Adapter or KEDA to expose `kaireon_decision_latency_ms` as a Kubernetes custom metric.
</details>

### HPA Tuning

**Adjusting target CPU utilization:**

```bash theme={null}
# Check current HPA status
kubectl get hpa kaireon-api-hpa -n kaireon

# View HPA events and scaling decisions
kubectl describe hpa kaireon-api-hpa -n kaireon

# Patch CPU target (lower = more aggressive scaling)
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"metrics":[{"type":"Resource","resource":{"name":"cpu","target":{"type":"Utilization","averageUtilization":50}}}]}}'
```

**Adjusting replica bounds:**

```bash theme={null}
# Increase max replicas for expected traffic spike
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"maxReplicas":30}}'

# Increase minimum replicas for guaranteed capacity
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"minReplicas":5}}'
```

**Adjusting scale-up/down behavior:**

```bash theme={null}
# Faster scale-up (for traffic spikes)
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"behavior":{"scaleUp":{"stabilizationWindowSeconds":30,"policies":[{"type":"Pods","value":6,"periodSeconds":30}]}}}}'

# Slower scale-down (prevent flapping)
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":600}}}}'
```

### Manual Scaling

For planned events or emergencies:

```bash theme={null}
# Scale to a specific number of replicas
kubectl scale deploy/kaireon-api -n kaireon --replicas=15

# Note: HPA will override manual scaling. To hold a fixed count,
# set minReplicas = maxReplicas temporarily:
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"minReplicas":15,"maxReplicas":15}}'

# Revert after the event
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"minReplicas":3,"maxReplicas":20}}'
```

### Pod Resource Tuning

```yaml theme={null}
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2000m"
    memory: "2Gi"
```

**Adjusting pod resources:**

```bash theme={null}
# Increase memory for OOM-prone pods
kubectl set resources deploy/kaireon-api -n kaireon \
  --requests=memory=1Gi \
  --limits=memory=4Gi

# Increase CPU for CPU-bound workloads
kubectl set resources deploy/kaireon-api -n kaireon \
  --requests=cpu=1000m \
  --limits=cpu=4000m
```

### Pre-scaling for Planned Events

```bash theme={null}
# 1. Scale up 30 minutes before the event
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"minReplicas":15}}'

# 2. Verify pods are ready
kubectl get pods -n kaireon -l app=kaireon-api | grep Running | wc -l

# 3. After the event, reduce minimum
kubectl patch hpa kaireon-api-hpa -n kaireon --type merge -p \
  '{"spec":{"minReplicas":3}}'
```

### Scheduler Behaviour When Running Multiple API Replicas

Each API pod starts two in-process schedulers on boot:

* The **Flow scheduler** (fires due pipelines and `file_arrival` triggers) takes
  a PostgreSQL advisory lock, so it is safe to run many API replicas — only the
  lock holder fires each tick. Disable with `FLOW_INTERNAL_SCHEDULER_ENABLED=false`.
* The **maintenance scheduler** (self-invokes the `/api/**/cron/*` jobs when
  `CRON_SECRET` is set) assumes a **single replica** — it is not
  distributed-locked, so N replicas run N duplicate passes. The cron jobs are
  idempotent, so duplicates are harmless, but for large fleets disable it on all
  replicas (`MAINTENANCE_SCHEDULER_ENABLED=false`) and drive the cron routes from
  one external scheduler (EventBridge or a K8s CronJob) instead.

***

## 2. Scaling Workers

### Current Architecture

```
Redis Queue (decisions, pipelines, batch)
    |
    v
KaireonAI Worker Deployment (M replicas, KEDA-managed)
```

### KEDA Autoscaler

The Helm chart ships a KEDA `ScaledObject` for the worker when
`worker.keda.enabled: true`. It scales the `kaireon-worker` deployment on the
depth of the BullMQ queue lists. **Chart defaults:** `minReplicas: 1`,
`maxReplicas: 10`, `queueThreshold: "5"` (see `helm/values.yaml`).

**Rendered ScaledObject (from `helm/templates/worker-deployment.yaml`):**

```yaml theme={null}
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kaireon-worker-scaler
  namespace: kaireon
spec:
  scaleTargetRef:
    name: kaireon-worker
  minReplicaCount: 1          # worker.keda.minReplicas
  maxReplicaCount: 10         # worker.keda.maxReplicas
  # pollingInterval / cooldownPeriod are not set by the chart and fall back
  # to KEDA defaults (30s poll, 300s cooldown).
  triggers:
    - type: redis
      metadata:
        address: redis.kaireon.svc.cluster.local:6379
        # BullMQ stores waiting jobs under bull:<queue>:wait
        listName: "bull:batch-jobs:wait"
        listLength: "5"        # worker.keda.queueThreshold
        enableTLS: "false"
        databaseIndex: "0"
```

The five worker queues are `batch-jobs`, `dsar-jobs`, `journey-jobs`,
`retrain-jobs`, and `seed-jobs`. Add a `redis` trigger per queue you want to
scale on, pointing `listName` at the matching `bull:<queue>:wait` key.

### KEDA Tuning

**Adjusting trigger thresholds:**

```bash theme={null}
# Edit the ScaledObject
kubectl edit scaledobject kaireon-worker-scaler -n kaireon

# Key parameters:
# - listLength: queue depth per replica (lower = more aggressive)
# - pollingInterval: how often KEDA checks (seconds)
# - cooldownPeriod: wait before scaling down (seconds)
```

**Recommended settings by scenario:**

| Scenario         | listLength | pollingInterval | cooldownPeriod | maxReplicas |
| ---------------- | ---------- | --------------- | -------------- | ----------- |
| Chart default    | 5          | 30              | 300            | 10          |
| High throughput  | 20         | 10              | 180            | 50          |
| Cost sensitive   | 100        | 30              | 600            | 15          |
| Batch processing | 10         | 10              | 60             | 50          |

Only `minReplicas`, `maxReplicas`, and `queueThreshold` (→ `listLength`) are
exposed through `helm/values.yaml`; the other columns require editing the
`ScaledObject` template or a custom overlay.

### Manual Worker Scaling

```bash theme={null}
# Temporarily override KEDA (pause the scaler)
kubectl annotate scaledobject kaireon-worker-scaler -n kaireon \
  autoscaling.keda.sh/paused-replicas="20"

# Scale directly
kubectl scale deploy/kaireon-worker -n kaireon --replicas=20

# Resume KEDA
kubectl annotate scaledobject kaireon-worker-scaler -n kaireon \
  autoscaling.keda.sh/paused-replicas-

# Or remove the annotation entirely
kubectl annotate scaledobject kaireon-worker-scaler -n kaireon \
  autoscaling.keda.sh/paused-replicas- --overwrite
```

### Worker Resource Tuning

Workers are CPU-intensive during scoring and memory-intensive during pipeline execution.

```yaml theme={null}
resources:
  requests:
    cpu: "1000m"
    memory: "1Gi"
  limits:
    cpu: "4000m"
    memory: "4Gi"
```

### Queue-Specific Worker Pools

For isolating workloads, deploy separate worker pools per queue:

```bash theme={null}
# Decision workers (latency-sensitive)
kubectl scale deploy/kaireon-worker-decisions -n kaireon --replicas=10

# Pipeline workers (throughput-focused)
kubectl scale deploy/kaireon-worker-pipelines -n kaireon --replicas=5

# Batch workers (cost-optimized, use spot instances)
kubectl scale deploy/kaireon-worker-batch -n kaireon --replicas=3
```

***

## 3. Scaling the Database

### Vertical Scaling (RDS)

**Instance type progression:**

| Tier   | Instance Type  | vCPUs | Memory | Max Connections | Use Case     |
| ------ | -------------- | ----- | ------ | --------------- | ------------ |
| Dev    | db.t3.medium   | 2     | 4 GB   | 100             | Development  |
| Small  | db.r6g.large   | 2     | 16 GB  | 200             | Low traffic  |
| Medium | db.r6g.xlarge  | 4     | 32 GB  | 400             | Standard     |
| Large  | db.r6g.2xlarge | 8     | 64 GB  | 800             | High traffic |
| XLarge | db.r6g.4xlarge | 16    | 128 GB | 1600            | Peak traffic |

**Scaling up:**

```bash theme={null}
# Check current instance type
aws rds describe-db-instances \
  --db-instance-identifier kaireon-prod \
  --query 'DBInstances[0].DBInstanceClass'

# Scale up (causes a brief outage during maintenance window)
aws rds modify-db-instance \
  --db-instance-identifier kaireon-prod \
  --db-instance-class db.r6g.2xlarge \
  --apply-immediately

# Monitor the modification
aws rds describe-db-instances \
  --db-instance-identifier kaireon-prod \
  --query 'DBInstances[0].[DBInstanceStatus,PendingModifiedValues]'
```

**Important:** Scaling up causes a brief outage (typically 1-3 minutes). Schedule during maintenance windows if possible. With Multi-AZ, failover minimizes downtime.

### Read Replicas

Use read replicas to offload read-heavy queries (dashboards, analytics, reporting).

**Creating a read replica:**

```bash theme={null}
aws rds create-db-instance-read-replica \
  --db-instance-identifier kaireon-prod-read-1 \
  --source-db-instance-identifier kaireon-prod \
  --db-instance-class db.r6g.xlarge \
  --availability-zone us-east-1b

# Wait for it to become available
aws rds wait db-instance-available --db-instance-identifier kaireon-prod-read-1
```

**Application configuration for read replicas:**

The application connects through a single `DATABASE_URL` and does **not** expose
a read/write-split env var (there is no `DATABASE_READ_URL`). To serve reads
from a replica, either point read-heavy tooling (dashboards, ad-hoc analytics,
BI) directly at the replica endpoint, or front reads with a proxy/PgBouncer
that routes to the replica. Routing the application's own reads to a replica
requires application-level changes, not a config toggle.

**Monitoring replica lag:**

```bash theme={null}
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name ReplicaLag \
  --dimensions Name=DBInstanceIdentifier,Value=kaireon-prod-read-1 \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Average
```

**Promoting a read replica (for failover or splitting):**

```bash theme={null}
aws rds promote-read-replica \
  --db-instance-identifier kaireon-prod-read-1
```

### Storage Scaling

```bash theme={null}
# Check current storage
aws rds describe-db-instances \
  --db-instance-identifier kaireon-prod \
  --query 'DBInstances[0].[AllocatedStorage,StorageType,Iops]'

# Increase storage (online, no downtime for gp3)
aws rds modify-db-instance \
  --db-instance-identifier kaireon-prod \
  --allocated-storage 500 \
  --apply-immediately

# Enable storage autoscaling
aws rds modify-db-instance \
  --db-instance-identifier kaireon-prod \
  --max-allocated-storage 1000 \
  --apply-immediately
```

### Connection Limit Scaling

When scaling the database vertically, adjust `max_connections` accordingly:

```bash theme={null}
# Check current max_connections
aws rds describe-db-parameters \
  --db-parameter-group-name kaireon-prod-params \
  --query "Parameters[?ParameterName=='max_connections']"

# RDS default formula: LEAST({DBInstanceClassMemory/9531392}, 5000)
# Override if needed:
aws rds modify-db-parameter-group \
  --db-parameter-group-name kaireon-prod-params \
  --parameters "ParameterName=max_connections,ParameterValue=400,ApplyMethod=pending-reboot"
```

***

## 4. Scaling Redis

### Vertical Scaling (ElastiCache)

**Instance type progression:**

| Tier   | Instance Type     | Memory   | Network       | Use Case      |
| ------ | ----------------- | -------- | ------------- | ------------- |
| Dev    | cache.t3.small    | 1.37 GB  | Low-Moderate  | Development   |
| Small  | cache.r6g.large   | 13.07 GB | Up to 10 Gbps | Standard      |
| Medium | cache.r6g.xlarge  | 26.32 GB | Up to 10 Gbps | High cache    |
| Large  | cache.r6g.2xlarge | 52.82 GB | Up to 10 Gbps | Large dataset |

**Scaling up (single node, non-clustered):**

```bash theme={null}
aws elasticache modify-replication-group \
  --replication-group-id kaireon-redis \
  --cache-node-type cache.r6g.xlarge \
  --apply-immediately
```

### Cluster Mode

For datasets that exceed single-node memory or require higher throughput.

**Enabling cluster mode:**

```bash theme={null}
# Create a new cluster-mode-enabled replication group
aws elasticache create-replication-group \
  --replication-group-id kaireon-redis-cluster \
  --replication-group-description "KaireonAI Redis Cluster" \
  --cache-node-type cache.r6g.large \
  --num-node-groups 3 \
  --replicas-per-node-group 1 \
  --automatic-failover-enabled \
  --multi-az-enabled \
  --cache-parameter-group-name default.redis7.cluster.on
```

**Resharding (adding shards):**

```bash theme={null}
aws elasticache modify-replication-group-shard-configuration \
  --replication-group-id kaireon-redis-cluster \
  --node-group-count 6 \
  --apply-immediately
```

**Application configuration for cluster mode:**

```bash theme={null}
# The app reads a single REDIS_URL; there is no REDIS_CLUSTER_MODE toggle.
# Point REDIS_URL at the cluster configuration endpoint — the ioredis client
# follows MOVED/ASK redirects. Validate BullMQ + rate-limiter behaviour under
# cluster mode before adopting it in production.
kubectl set env deploy/kaireon-api \
  REDIS_URL="redis://kaireon-redis-cluster.xxxxxxxx.clustercfg.use1.cache.amazonaws.com:6379"
```

### Redis Memory Management

```bash theme={null}
# Check memory usage
kubectl exec -it deploy/kaireon-redis -- redis-cli INFO memory

# Set maxmemory
kubectl exec -it deploy/kaireon-redis -- redis-cli CONFIG SET maxmemory 12gb

# Set eviction policy
kubectl exec -it deploy/kaireon-redis -- redis-cli CONFIG SET maxmemory-policy volatile-lru
```

### Scaling Redis for Specific Use Cases

**Decision caching (read-heavy):**

* Use read replicas for read distribution.
* Set short TTLs (30-60s) to limit memory growth.
* Use `volatile-lru` eviction.

**Queue processing (write-heavy):**

* Scale vertically for more throughput.
* Monitor `instantaneous_ops_per_sec`.
* Consider cluster mode if >100K ops/sec.

**Session storage:**

* Separate from cache Redis.
* Use `noeviction` policy (sessions must not be evicted).
* Size for peak concurrent users.

***

## 5. Scaling PgBouncer

### Pool Size Calculations

```
Total server connections needed:
  = (API replicas * connections_per_pod) + (Worker replicas * connections_per_worker)
  + monitoring + replication + admin overhead

Example:
  API: 10 replicas * 5 connections = 50
  Workers: 10 replicas * 3 connections = 30
  Monitoring: 5
  Replication: 5
  Admin: 10
  Total: 100

PgBouncer settings:
  max_db_connections = 100 (per PgBouncer instance)
  default_pool_size = 50 (per database)
  max_client_conn = 500 (per PgBouncer instance)
```

### Scaling PgBouncer Horizontally

Deploy multiple PgBouncer instances behind a Kubernetes Service:

```bash theme={null}
# Scale PgBouncer replicas
kubectl scale deploy/kaireon-pgbouncer -n kaireon --replicas=3

# Verify all replicas are healthy
kubectl get pods -n kaireon -l app=kaireon-pgbouncer
```

**Adjusting pool size when scaling API/workers:**

| API Replicas | Worker Replicas | PgBouncer Instances | default\_pool\_size | max\_db\_connections |
| ------------ | --------------- | ------------------- | ------------------- | -------------------- |
| 3            | 2               | 1                   | 25                  | 80                   |
| 10           | 10              | 2                   | 25                  | 100                  |
| 20           | 20              | 3                   | 30                  | 120                  |
| 30           | 30              | 4                   | 25                  | 100                  |

**Important:** Total `max_db_connections` across all PgBouncer instances must not exceed PostgreSQL `max_connections`.

### Applying Pool Changes

```bash theme={null}
# Edit ConfigMap
kubectl edit configmap kaireon-pgbouncer-config -n kaireon

# Hot-reload (no restart needed for pool size changes)
for pod in $(kubectl get pods -n kaireon -l app=kaireon-pgbouncer -o name); do
  kubectl exec -n kaireon $pod -- psql -p 6432 pgbouncer -c "RELOAD;"
done

# Verify new settings
kubectl exec -it deploy/kaireon-pgbouncer -n kaireon -- psql -p 6432 pgbouncer -c "SHOW CONFIG;" | grep pool
```

***

## 6. Capacity Planning

### Metrics to Track

| Metric                 | Current | Warning Threshold | Scaling Action            |
| ---------------------- | ------- | ----------------- | ------------------------- |
| API CPU utilization    | -       | >60% sustained    | Increase HPA maxReplicas  |
| API memory utilization | -       | >75% sustained    | Increase pod memory limit |
| Worker queue depth     | -       | >50 sustained     | Lower KEDA listLength     |
| DB CPU utilization     | -       | >70% sustained    | Scale up instance type    |
| DB connections used    | -       | >70% of max       | Scale PgBouncer or DB     |
| DB storage used        | -       | >70% of allocated | Increase storage          |
| Redis memory used      | -       | >70% of maxmemory | Scale up instance type    |
| Redis ops/sec          | -       | >80% of benchmark | Enable cluster mode       |

### Monthly Capacity Review Checklist

1. Review 30-day trends for all metrics above.
2. Project growth for the next 90 days based on customer onboarding pipeline.
3. Identify any component within 30 days of hitting a threshold.
4. Plan scaling actions with cost estimates.
5. Update this document with new current values.

### Cost-Aware Scaling

* Use **Spot instances** for batch workers (up to 70% savings).
* Use **Reserved Instances** for baseline API and database capacity.
* Use **Graviton (arm64)** instances for 20% better price-performance.
* Scale down non-production environments during off-hours:
  ```bash theme={null}
  # Scale down staging at 8 PM
  kubectl scale deploy -n kaireon-staging --all --replicas=0

  # Scale up at 8 AM
  kubectl scale deploy -n kaireon-staging --all --replicas=1
  ```

### Load Testing Before Scaling

Before major scaling changes, validate with load tests:

```bash theme={null}
# Install k6
brew install k6

# Run a decision endpoint load test
k6 run --vus 100 --duration 5m scripts/load-test-decisions.js

# Run with ramping to find the breaking point
k6 run --stage '1m:50,3m:200,1m:50' scripts/load-test-decisions.js
```

Key metrics to capture during load tests:

* P50, P95, P99 latency at each VU level.
* Error rate at each VU level.
* Database connection count and query time.
* Redis memory and ops/sec.
* Pod CPU and memory utilization.