Scaling & Performance

​

Request Lifecycle

​

Caching Strategy

​

What gets cached

​

Tuning TTLs

{
  "enrichment": {
    "sources": [
      {
        "schemaId": "customer_profile",
        "fields": ["loan_amount", "credit_score"],
        "cacheTtlSeconds": 300,
        "prefix": "customer"
      },
      {
        "schemaId": "real_time_signals",
        "fields": ["last_page_viewed"],
        "cacheTtlSeconds": 10,
        "prefix": "signals"
      }
    ]
  }
}

​

Cache Invalidation

​

Rate Limiting

​

Algorithm

1. ZREMRANGEBYSCORE — remove entries outside the current window
2. ZADD — add the current request with its timestamp as score
3. ZCARD — count entries remaining in the window
4. EXPIRE — set key TTL to window duration + 1 second for cleanup

​

Tier Configuration

# Global default tier
RATE_LIMIT_TIER=standard

# Per-tenant override (tenant ID uppercased, hyphens to underscores)
RATE_LIMIT_TIER_ACME_CORP=enterprise

​

Response Headers

​

Fail-Open vs Fail-Closed

• Fail-open (default): Falls back to in-memory rate limiting. Use for standard API endpoints where availability matters more than strict enforcement.
• Fail-closed: Returns 429 when Redis is down. The Recommend API uses failOpen: false to prevent abuse when rate limit state is unavailable.

​

Edge-Layer Rate Limiting

​

Circuit Breakers

​

Parameters

​

Behavior

1. Each scoring model is tracked by its modelKey.
2. On a model error, the failure counter increments via recordModelFailure().
3. When failures reach the threshold (5), the circuit opens and sets a cooldown expiry at now + 60 seconds.
4. While open, all requests for that model skip the model call entirely and receive the fallback score (0.5 * weight * fitMultiplier).
5. After the cooldown expires, the next request acts as a probe — if it succeeds, recordModelSuccess() resets the counter (circuit closes). If it fails, the circuit re-opens for another cooldown period.
6. The response includes degradedScoring: true when any model was bypassed.

​

Monitoring

​

Atomic Cap Checking

1. INCR kaireon:cap:mandatory:{customerId}:{YYYY-MM-DD} — single atomic read-and-increment
2. On first increment (current === 1), set EXPIRE to end of current UTC day
3. If current > cap, the offer is blocked
4. Default cap: 5 mandatory offers per customer per day (configurable via MAX_MANDATORY_OFFERS_PER_DAY env var)

​

Connection Pooling

​

Configuration

# Example with pool sizing parameters
DATABASE_URL="postgresql://user:pass@host:5432/kaireon?connection_limit=20&pool_timeout=10"

​

Sizing Guidance

​

Horizontal Scaling

​

Architecture

                    ┌─────────────┐
                    │   Load      │
                    │  Balancer   │
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
        ┌─────┴─────┐ ┌───┴─────┐ ┌───┴─────┐
        │  API Pod   │ │ API Pod │ │ API Pod │
        │  (Node.js) │ │         │ │         │
        └─────┬──────┘ └───┬─────┘ └───┬─────┘
              │            │            │
              └────────────┼────────────┘
                           │
              ┌────────────┼────────────┐
              │                         │
        ┌─────┴─────┐           ┌──────┴──────┐
        │   Redis   │           │ PostgreSQL  │
        │ (shared)  │           │  (shared)   │
        └───────────┘           └─────────────┘

​

Key Properties

• No session affinity required: Any API pod can handle any request. Rate limit state and caching are in Redis; all persistent state is in PostgreSQL.
• In-memory circuit breakers are per-process: Each pod tracks its own model failure counts. This is intentional — a model failure on one pod does not cascade to others, and each pod independently probes recovery.
• In-memory rate limit fallback is per-process: When Redis is down, each pod maintains its own rate limit counters. Effective limits become configured_limit x num_pods during Redis outages.
• Scale API pods independently from worker pods: Decision API pods handle synchronous request/response. Data pipeline worker pods handle asynchronous ETL. Size each tier based on its workload.

​

Batch vs Streaming Pipelines

​

Batch Mode

{
  "mode": "batch",
  "batchSize": 1000,
  "parallelism": 4,
  "partitioning": {
    "strategy": "hash",
    "key": "customer_id",
    "partitions": 8
  }
}

​

Streaming Mode

{
  "mode": "streaming",
  "batchSize": 100,
  "parallelism": 2,
  "checkpointIntervalMs": 30000
}

​

K8s Worker Pod Configuration

• CPU-bound transforms (expression evaluation, hashing): Scale parallelism up to available CPU cores.
• I/O-bound transforms (external lookups, PII masking): Higher parallelism with moderate CPU allocation.
• Memory-bound transforms (large batch joins): Increase pod memory limits and reduce batchSize.

​

Production Tuning Checklist

​

Under 1K decisions/day

• Single API instance with default settings
• In-memory rate limiting and caching are sufficient
• Default connection pool (5-10 connections)
• No Redis required (in-memory fallbacks handle the load)

​

1K — 100K decisions/day

• Redis required for rate limiting, enrichment caching, and atomic cap checks
• Tune enrichment TTLs: increase to 300s+ for stable customer data
• Increase connection pool to 15-20 per instance
• Enable decision tracing with a sample rate (e.g., 10%) rather than 100%
• Monitor decisionLatencyMs and scoringModelFailureTotal metrics

​

100K+ decisions/day

• Multiple API replicas behind a load balancer (3+ pods recommended)
• Dedicated Redis instance (ElastiCache or equivalent) with sufficient memory for rate limit sorted sets + enrichment cache
• PostgreSQL read replicas for offer/policy reads; primary for writes only
• Use PgBouncer or RDS Proxy to multiplex database connections
• Set MAX_ACTIVE_OFFERS env var to limit the offer scan set (default: 5,000)
• Lower guardrail TTL if policy changes need sub-30s propagation
• Configure edge-layer rate limiting (AWS WAF, Cloudflare) for DDoS protection
• Enable budget pacing for high-volume offers to spread delivery across the day
• Set SCORING_FALLBACK_SCORE to a value appropriate for your scoring distribution

​

Environment Variables Reference

​

Related