Incident Response

Audience: On-call engineers, SREs, platform operators Last updated: 2026-02-23 Escalation chain: On-call engineer -> Team lead -> VP Engineering -> CTO

​

Table of Contents

1. Decision Latency Spike (>200ms P99)
2. Worker Backlog (Queue Depth >100)
3. Database Connection Exhaustion
4. Redis OOM / Eviction
5. API 5xx Error Rate >1%
6. DLQ Overflow (>500)
7. Certificate Expiry
8. Node Capacity Exhaustion
9. Database Disk Full
10. Auth Failures Spike

​

General Incident Workflow

1. Acknowledge the alert within 5 minutes.
2. Assess severity using the matrix below.
3. Communicate status in #kaireon-incidents Slack channel.
4. Mitigate using the relevant playbook.
5. Resolve and confirm metrics return to normal.
6. Post-mortem within 48 hours for SEV1/SEV2 incidents.

​

1. Decision Latency Spike (>200ms P99)

​

Detection

• Alert: kaireon_decision_latency_ms P99 > 200ms for 3 consecutive minutes.
• Dashboard: Grafana > KaireonAI Decisions > Latency panel.
• Metrics:
  histogram_quantile(0.99, rate(kaireon_decision_latency_ms_bucket[5m]))
  

​

Diagnosis

1. Identify the bottleneck layer:
   # Check if the database is slow
   kubectl exec -it deploy/kaireon-api -- curl -s localhost:3000/api/metrics | grep kaireon_http_request_duration_seconds
   
   # Check Redis latency
   kubectl exec -it deploy/kaireon-redis -- redis-cli --latency-history -i 1
   
   # Check scoring engine time
   kubectl logs deploy/kaireon-api --since=5m | grep "scoring_duration" | tail -20
   
2. Check for resource contention:
   # CPU throttling on decision pods
   kubectl top pods -l app=kaireon-api
   
   # Check if HPA is maxed out
   kubectl get hpa kaireon-api-hpa
   
3. Check for slow queries:
   SELECT pid, now() - pg_stat_activity.query_start AS duration, query
   FROM pg_stat_activity
   WHERE state != 'idle'
   AND now() - pg_stat_activity.query_start > interval '1 second'
   ORDER BY duration DESC;
   
4. Check for model loading delays:
   kubectl logs deploy/kaireon-api --since=5m | grep -E "model_load|cache_miss"
   

​

Resolution

1. Immediate (if >500ms P99): Scale up API replicas:
   kubectl scale deploy/kaireon-api --replicas=10
   
2. If database-bound: Kill long-running queries and investigate:
   SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state != 'idle'
   AND now() - pg_stat_activity.query_start > interval '30 seconds'
   AND query NOT LIKE '%pg_stat_activity%';
   
3. If Redis-bound: Flush stale caches:
   kubectl exec -it deploy/kaireon-redis -- redis-cli FLUSHDB ASYNC
   
4. If scoring-bound: identify the slow model via kaireon_scoring_latency_ms (and check kaireon_scoring_model_failure_total for failing models), then add API capacity per step 1. Scoring has no runtime env-var toggle; tune the model configuration in the Algorithms module instead.

​

Prevention

• Set HPA target CPU to 60% (not 80%) for headroom.
• Enable scoring result caching with a 30-second TTL for repeat customer lookups.
• Run weekly load tests against the decision endpoint with production-like traffic.
• Maintain database query performance baselines; alert on 50% regression.
• Pre-warm model caches on pod startup using an init container.

​

2. Worker Backlog (Queue Depth >100)

​

Detection

• Alert: kaireon_active_worker_jobs > 100 for 5 minutes.
• Dashboard: Grafana > KaireonAI Workers > Queue Depth panel.
• Metrics:
  kaireon_active_worker_jobs{queue="batch-jobs"}
  rate(kaireon_active_worker_jobs[5m])
  
• Queue names: batch-jobs, dsar-jobs, journey-jobs, retrain-jobs, seed-jobs (the five BullMQ queues). Decision requests via /api/v1/recommend are served synchronously and do not flow through a worker queue.

​

Diagnosis

1. Determine which queue is backed up:
   kubectl exec -it deploy/kaireon-api -- curl -s localhost:3000/api/metrics | grep kaireon_active_worker_jobs
   
2. Check worker health:
   kubectl get pods -l app=kaireon-worker
   kubectl logs -l app=kaireon-worker --since=5m --tail=50 | grep -E "error|panic|OOM"
   
3. Check if workers are stuck on a specific job:
   kubectl logs -l app=kaireon-worker --since=5m | grep "processing_duration" | sort -t= -k2 -rn | head -10
   
4. Check for upstream rate changes:
   rate(kaireon_worker_jobs_enqueued_total[5m])
   

​

Resolution

1. Scale workers immediately:
   kubectl scale deploy/kaireon-worker --replicas=20
   
2. If workers are crash-looping, restart them:
   kubectl rollout restart deploy/kaireon-worker
   
3. If a single queue is stuck, isolate it: the worker has no per-queue skip env var. Pause the offending queue via BullMQ tooling (e.g. Bull Board), or restart the workers to clear a hung job:
   kubectl rollout restart deploy/kaireon-worker
   
4. If the queue is Redis-backed and Redis is the bottleneck:
   kubectl exec -it deploy/kaireon-redis -- redis-cli INFO memory
   kubectl exec -it deploy/kaireon-redis -- redis-cli SLOWLOG GET 10
   

​

Prevention

• Configure KEDA autoscaler to trigger at queue depth 50 (not 100).
• Set per-job-type timeouts (default 60s) so stuck jobs do not block workers.
• Implement circuit breakers for downstream dependencies (connectors, external APIs).
• Monitor queue depth trend, not just threshold; alert on sustained growth rate.
• Set max retry count to 3 with exponential backoff to prevent retry storms.

​

3. Database Connection Exhaustion

​

Detection

• Alert: pgbouncer_active_connections / pgbouncer_max_connections > 0.9 for 2 minutes.
• Alert: Application logs contain remaining connection slots are reserved or too many connections.
• Dashboard: Grafana > PostgreSQL > Connection Pool panel.
• Metrics:
  pgbouncer_pools_server_active_connections
  pgbouncer_pools_server_idle_connections
  pgbouncer_pools_client_waiting_connections
  

​

Diagnosis

1. Check PgBouncer status:
   kubectl exec -it deploy/kaireon-pgbouncer -- psql -p 6432 pgbouncer -c "SHOW POOLS;"
   kubectl exec -it deploy/kaireon-pgbouncer -- psql -p 6432 pgbouncer -c "SHOW CLIENTS;"
   
2. Check for connection leaks in the application:
   kubectl logs -l app=kaireon-api --since=10m | grep -E "connection|pool" | tail -30
   
3. Check PostgreSQL directly:
   SELECT count(*), state, usename, application_name
   FROM pg_stat_activity
   GROUP BY state, usename, application_name
   ORDER BY count DESC;
   
4. Check for long-running transactions holding connections:
   SELECT pid, now() - xact_start AS duration, state, query
   FROM pg_stat_activity
   WHERE xact_start IS NOT NULL
   ORDER BY xact_start ASC
   LIMIT 20;
   

​

Resolution

1. Kill idle-in-transaction connections (>5 min):
   SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state = 'idle in transaction'
   AND now() - xact_start > interval '5 minutes';
   
2. Reload PgBouncer if it is stuck:
   kubectl exec -it deploy/kaireon-pgbouncer -- psql -p 6432 pgbouncer -c "RELOAD;"
   
3. Temporarily increase PgBouncer pool size:
   kubectl edit configmap kaireon-pgbouncer-config
   # Increase default_pool_size from 20 to 40
   kubectl rollout restart deploy/kaireon-pgbouncer
   
4. If a specific service is leaking, restart it:
   kubectl rollout restart deploy/kaireon-api
   

​

Prevention

• Set idle_in_transaction_session_timeout = 30s in PostgreSQL.
• Configure PgBouncer server_idle_timeout = 600 to reclaim idle server connections.
• Use connection pooling in the application layer (Prisma connection limit).
• Set max_client_conn in PgBouncer to 2x expected peak.
• Add connection acquisition timeout (5s) in the application to fail fast.
• Audit code for missing finally blocks that release connections.

​

4. Redis OOM / Eviction or Quota Exhaustion

Free-tier quota note: On Upstash / Fly Redis / similar pay-by-ops services, this incident may present as ERR max requests limit exceeded. Limit: 500000 rather than OOM. The most common cause is idle BullMQ worker polling with WORKER_INPROCESS=1 — five workers polling Redis with the BRPOPLPUSH command burn ~1.3M ops/month with zero queued work. The fix is WORKER_INPROCESS=0 + cron-driven drain; see worker-mode-and-cron-drain runbook §8 for the full diagnosis + resolution. Real /recommend traffic on a typical playground is ~75-120K ops/month — well within the 500K free tier if the worker isn’t burning it idle.

​

Detection

• Alert: redis_memory_used_bytes / redis_memory_max_bytes > 0.9 for 5 minutes.
• Alert: redis_evicted_keys_total rate > 0 for 3 minutes.
• Dashboard: Grafana > Redis > Memory panel.
• Metrics:
  redis_memory_used_bytes
  rate(redis_evicted_keys_total[5m])
  redis_connected_clients
  

​

Diagnosis

1. Check memory breakdown:
   kubectl exec -it deploy/kaireon-redis -- redis-cli INFO memory
   kubectl exec -it deploy/kaireon-redis -- redis-cli MEMORY DOCTOR
   
2. Identify largest keys:
   kubectl exec -it deploy/kaireon-redis -- redis-cli --bigkeys
   
3. Check which key patterns consume the most memory:
   kubectl exec -it deploy/kaireon-redis -- redis-cli --memkeys --pattern "decision:*" | tail -5
   kubectl exec -it deploy/kaireon-redis -- redis-cli --memkeys --pattern "session:*" | tail -5
   
4. Check for abnormal client connections:
   kubectl exec -it deploy/kaireon-redis -- redis-cli CLIENT LIST | wc -l
   

​

Resolution

1. Flush non-critical caches first:
   # Flush only the cache database (DB 1), not the session database (DB 0)
   kubectl exec -it deploy/kaireon-redis -- redis-cli -n 1 FLUSHDB ASYNC
   
2. If specific key patterns are bloated, expire them:
   kubectl exec -it deploy/kaireon-redis -- redis-cli EVAL "
     local keys = redis.call('KEYS', ARGV[1])
     for _, key in ipairs(keys) do
       if redis.call('TTL', key) == -1 then
         redis.call('EXPIRE', key, 300)
       end
     end
     return #keys
   " 0 "pipeline:result:*"
   
3. Scale Redis vertically (if managed):
   # For AWS ElastiCache — change instance type via console or Terraform
   # For self-managed, increase maxmemory:
   kubectl exec -it deploy/kaireon-redis -- redis-cli CONFIG SET maxmemory 8gb
   
4. Switch eviction policy if needed:
   kubectl exec -it deploy/kaireon-redis -- redis-cli CONFIG SET maxmemory-policy allkeys-lru
   

​

Prevention

• Set TTLs on all cache keys (decision cache: 60s, session: 24h, feature: 300s).
• Use volatile-lru eviction policy so only keys with TTLs are evicted.
• Monitor memory usage trend and scale proactively at 70%.
• Separate cache Redis from session/queue Redis to isolate failure domains.
• Implement key size limits in the application layer (reject values >1MB).

​

5. API 5xx Error Rate >1%

​

Detection

• Alert: rate(http_responses_total{status=~"5.."}[5m]) / rate(http_responses_total[5m]) > 0.01 for 3 minutes.
• Dashboard: Grafana > KaireonAI API > Error Rate panel.
• Metrics:
  sum(rate(http_responses_total{status=~"5.."}[5m])) by (route)
  / sum(rate(http_responses_total[5m])) by (route)
  

​

Diagnosis

1. Identify which endpoints are failing:
   topk(5, sum(rate(http_responses_total{status=~"5.."}[5m])) by (route))
   
2. Check application logs for errors:
   kubectl logs -l app=kaireon-api --since=5m | grep -E '"status":5[0-9]{2}' | head -20
   kubectl logs -l app=kaireon-api --since=5m | grep -i "error\|exception\|panic" | tail -30
   
3. Check if it correlates with a deployment:
   kubectl rollout history deploy/kaireon-api
   
4. Check downstream dependencies:
   # Database connectivity
   kubectl exec -it deploy/kaireon-api -- curl -s localhost:3000/api/health
   
   # Redis connectivity
   kubectl exec -it deploy/kaireon-redis -- redis-cli PING
   
5. Check for resource pressure:
   kubectl top pods -l app=kaireon-api
   kubectl describe pods -l app=kaireon-api | grep -A5 "Last State"
   

​

Resolution

1. If caused by a bad deployment, rollback:
   kubectl rollout undo deploy/kaireon-api
   
2. If caused by a downstream failure: the circuit breakers are always on — watch kaireon_circuit_breaker_state_change_total to confirm the breaker for the failing dependency has tripped, and wait for it to half-open once the dependency recovers.
3. If caused by OOM kills, increase memory:
   kubectl set resources deploy/kaireon-api --limits=memory=2Gi --requests=memory=1Gi
   
4. If a specific route is the problem: there is no application route-blocklist env var. Block the route at the ingress/ALB (or roll back the deploy that introduced the regression) while you investigate.

​

Prevention

• Implement canary deployments (10% traffic for 5 minutes before full rollout).
• Add structured error logging with request IDs for traceability.
• Set memory limits with 20% headroom above observed peak.
• Run integration tests in staging before promoting to production.
• Implement retry with backoff for transient downstream failures.

​

6. DLQ Overflow (>500)

The DLQ is a single BullMQ queue named dead-letter. A job from any of the five source queues (batch-jobs, dsar-jobs, journey-jobs, retrain-jobs, seed-jobs) is moved there after it exhausts its retries. Decision requests are served synchronously by /api/v1/recommend and are not queued, so they never appear in the DLQ.

​

Detection

• Alert: kaireon_dlq_depth > 500 for 10 minutes.
• Dashboard: Grafana > KaireonAI Workers > Dead Letter Queue panel.
• Metrics (the kaireon_dlq_depth gauge is labelled by tenant):
  kaireon_dlq_depth
  rate(kaireon_dlq_depth[5m])
  

​

Diagnosis

1. Measure the dead-letter queue depth directly:
   # BullMQ keeps waiting jobs under bull:<queue>:wait
   kubectl exec -it deploy/kaireon-redis -- redis-cli LLEN bull:dead-letter:wait
   
2. Find which failures dominate (the worker logs Moved job to dead-letter queue with the originating sourceQueue):
   kubectl logs -l app=kaireon-worker --since=30m | grep "Moved job to dead-letter queue" | \
     jq -r '.sourceQueue' | sort | uniq -c | sort -rn
   
3. Check if the DLQ growth correlates with a deployment or config change:
   kubectl rollout history deploy/kaireon-worker
   kubectl get events --sort-by=.lastTimestamp | tail -20
   

​

Resolution

1. If the failures were transient (downstream outage now recovered): confirm the dependency is healthy, then re-enqueue the dead-letter jobs onto their source queue in controlled batches with BullMQ.
   # Verify downstream is healthy first
   kubectl exec -it deploy/kaireon-api -- curl -s localhost:3000/api/health
   
2. If the messages are poison pills (permanent schema/validation errors): export the payloads for forensics, then clear the waiting list.
   # Blunt purge of the dead-letter waiting list (drops the jobs)
   kubectl exec -it deploy/kaireon-redis -- redis-cli DEL bull:dead-letter:wait
   
3. If a bad deploy caused the spike: roll back the worker, which stops new jobs entering the DLQ, then re-drive the accumulated jobs once the fix ships.
   kubectl rollout undo deploy/kaireon-worker
   

​

Prevention

• Set DLQ alert threshold at 100 (not 500) for earlier detection.
• Implement automatic DLQ replay with exponential backoff (max 3 retries).
• Add DLQ message classification (transient vs. permanent failure).
• Archive DLQ messages to S3 daily for audit and forensics.
• Add schema validation before enqueue to reject malformed messages early.

​

7. Certificate Expiry

​

Detection

• Alert: cert_expiry_seconds < 604800 (7 days) for warning.
• Alert: cert_expiry_seconds < 86400 (24 hours) for critical.
• Dashboard: Grafana > Infrastructure > Certificate Expiry panel.
• Manual check:
  # Check TLS cert for the API endpoint
  echo | openssl s_client -connect api.kaireon.com:443 2>/dev/null | openssl x509 -noout -dates
  
  # Check all Kubernetes TLS secrets
  kubectl get secrets --all-namespaces -o json | \
    jq -r '.items[] | select(.type=="kubernetes.io/tls") |
    "\(.metadata.namespace)/\(.metadata.name)"' | while read secret; do
      ns=$(echo $secret | cut -d/ -f1)
      name=$(echo $secret | cut -d/ -f2)
      expiry=$(kubectl get secret -n $ns $name -o jsonpath='{.data.tls\.crt}' | \
        base64 -d | openssl x509 -noout -enddate 2>/dev/null)
      echo "$secret: $expiry"
    done
  

​

Diagnosis

1. Identify which certificate is expiring:
   kubectl get certificate --all-namespaces
   kubectl describe certificate -n kaireon kaireon-tls
   
2. Check cert-manager status (if using cert-manager):
   kubectl get challenges --all-namespaces
   kubectl get orders --all-namespaces
   kubectl logs -n cert-manager deploy/cert-manager --since=1h | grep -i error
   
3. Check if auto-renewal failed:
   kubectl describe certificaterequest -n kaireon | grep -A5 "Status"
   

​

Resolution

1. If cert-manager is installed, force renewal:
   kubectl cert-manager renew -n kaireon kaireon-tls
   
2. If cert-manager renewal is stuck, delete and recreate:
   kubectl delete certificate -n kaireon kaireon-tls
   kubectl apply -f k8s/certificates/kaireon-tls.yaml
   
3. If manual certificate, replace it:
   kubectl create secret tls kaireon-tls \
     --cert=new-cert.pem \
     --key=new-key.pem \
     -n kaireon \
     --dry-run=client -o yaml | kubectl apply -f -
   
   # Restart ingress controller to pick up new cert
   kubectl rollout restart deploy/ingress-nginx-controller -n ingress-nginx
   

​

Prevention

• Use cert-manager with Let’s Encrypt for automatic renewal.
• Set alerts at 30, 14, 7, 3, and 1 day(s) before expiry.
• Run a weekly certificate audit job that scans all namespaces.
• Maintain a certificate inventory spreadsheet with owners and expiry dates.
• Test certificate renewal in staging monthly.

​

8. Node Capacity Exhaustion

​

Detection

• Alert: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1.
• Alert: kube_node_status_condition{condition="DiskPressure",status="true"} == 1.
• Alert: Pending pods count > 0 for more than 5 minutes.
• Dashboard: Grafana > Kubernetes > Node Resources panel.
• Metrics:
  sum(kube_pod_status_phase{phase="Pending"}) > 0
  node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
  

​

Diagnosis

1. Check node resource usage:
   kubectl top nodes
   kubectl describe nodes | grep -A10 "Allocated resources"
   
2. Check for pending pods:
   kubectl get pods --all-namespaces --field-selector=status.phase=Pending
   kubectl describe pod <pending-pod-name> | grep -A5 "Events"
   
3. Check for resource hogs:
   kubectl top pods --all-namespaces --sort-by=memory | head -20
   kubectl top pods --all-namespaces --sort-by=cpu | head -20
   
4. Check cluster autoscaler status:
   kubectl logs -n kube-system deploy/cluster-autoscaler --since=10m | grep -E "scale|unschedulable"
   

​

Resolution

1. If autoscaler is enabled, check if it is working:
   kubectl get configmap -n kube-system cluster-autoscaler-status -o yaml
   
2. Manually add nodes if autoscaler is stuck:
   # For EKS
   aws eks update-nodegroup-config \
     --cluster-name kaireon-prod \
     --nodegroup-name kaireon-workers \
     --scaling-config minSize=3,maxSize=20,desiredSize=10
   
3. Evict non-critical workloads:
   kubectl scale deploy/kaireon-batch-processor --replicas=0
   kubectl scale deploy/kaireon-analytics --replicas=0
   
4. If disk pressure, clean up:
   # On the affected node
   docker system prune -af
   crictl rmi --prune
   

​

Prevention

• Set cluster autoscaler min/max to allow 30% headroom.
• Use PodDisruptionBudgets to protect critical workloads during eviction.
• Set resource requests and limits on all pods (no unbounded pods).
• Schedule non-critical batch jobs during off-peak hours.
• Run monthly capacity planning reviews based on growth trends.

​

9. Database Disk Full

​

Detection

• Alert: pg_database_size_bytes / pg_disk_total_bytes > 0.85 for 10 minutes.
• Alert (RDS): FreeStorageSpace < 5GB for 5 minutes.
• Dashboard: Grafana > PostgreSQL > Disk Usage panel.
• Metrics:
  pg_database_size_bytes{datname="kaireon"}
  node_filesystem_avail_bytes{mountpoint="/var/lib/postgresql"}
  

​

Diagnosis

1. Check current disk usage:
   SELECT pg_size_pretty(pg_database_size('kaireon')) AS db_size;
   
   SELECT schemaname, tablename,
     pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size
   FROM pg_tables
   WHERE schemaname = 'public'
   ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC
   LIMIT 20;
   
2. Check for bloat:
   SELECT schemaname, tablename,
     pg_size_pretty(pg_relation_size(schemaname || '.' || tablename)) AS table_size,
     n_dead_tup,
     n_live_tup,
     ROUND(n_dead_tup::numeric / NULLIF(n_live_tup, 0) * 100, 2) AS dead_pct
   FROM pg_stat_user_tables
   ORDER BY n_dead_tup DESC
   LIMIT 20;
   
3. Check for WAL accumulation:
   SELECT pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0')) AS wal_size;
   SELECT * FROM pg_replication_slots;
   
4. Check for orphaned temp files:
   SELECT pg_size_pretty(temp_bytes) AS temp_usage, datname
   FROM pg_stat_database
   WHERE temp_bytes > 0;
   

​

Resolution

1. Immediate: Increase disk (if RDS with storage autoscaling disabled):
   aws rds modify-db-instance \
     --db-instance-identifier kaireon-prod \
     --allocated-storage 500 \
     --apply-immediately
   
2. Run emergency VACUUM on bloated tables:
   VACUUM (VERBOSE) decision_logs;
   VACUUM (VERBOSE) audit_events;
   
3. Purge old data (if retention policy allows):
   DELETE FROM decision_logs WHERE created_at < NOW() - INTERVAL '90 days';
   DELETE FROM audit_events WHERE created_at < NOW() - INTERVAL '180 days';
   VACUUM FULL decision_logs;
   
4. Drop unused replication slots (WAL accumulation):
   SELECT pg_drop_replication_slot('unused_slot_name');
   

​

Prevention

• Enable RDS storage autoscaling with a maximum limit.
• Implement data retention policies with automated purge jobs (daily cron).
• Run VACUUM ANALYZE on large tables nightly.
• Partition large tables (decision_logs, audit_events) by month.
• Monitor disk growth rate and project exhaustion date weekly.

​

10. Auth Failures Spike

​

Detection

• Alert: rate(kaireon_auth_failures_total[5m]) > 10 for 3 minutes.
• Alert: rate(kaireon_auth_failures_total[5m]) / rate(kaireon_auth_attempts_total[5m]) > 0.1 for 5 minutes.
• Dashboard: Grafana > KaireonAI Security > Auth Failures panel.
• Metrics:
  rate(kaireon_auth_failures_total[5m])
  topk(5, sum(rate(kaireon_auth_failures_total[5m])) by (reason))
  topk(5, sum(rate(kaireon_auth_failures_total[5m])) by (source_ip))
  

​

Diagnosis

1. Identify the failure reason:
   kubectl logs -l app=kaireon-api --since=10m | grep "auth_failure" | \
     jq -r '.reason' | sort | uniq -c | sort -rn
   
2. Check if failures are from a single IP (brute force):
   kubectl logs -l app=kaireon-api --since=10m | grep "auth_failure" | \
     jq -r '.source_ip' | sort | uniq -c | sort -rn | head -10
   
3. Check if OIDC provider is down:
   curl -s https://auth.kaireon.com/.well-known/openid-configuration | jq .
   curl -s https://auth.kaireon.com/health
   
4. Check if tokens are expired due to clock skew:
   # Check API server time
   kubectl exec -it deploy/kaireon-api -- date -u
   
   # Compare with auth server
   curl -sI https://auth.kaireon.com | grep Date
   
5. Check for recent secret/key rotation:
   kubectl get secret kaireon-auth-config -o yaml | grep -E "last-applied|resourceVersion"
   

​

Resolution

1. If brute force, block the source IP:
   # Add to WAF blocklist
   aws wafv2 update-ip-set \
     --name kaireon-blocklist \
     --scope REGIONAL \
     --addresses "1.2.3.4/32" \
     --id <ip-set-id> \
     --lock-token <lock-token>
   
2. If the OIDC/SSO provider is down: there is no fallback-auth env toggle. Service-to-service traffic authenticated with API keys is unaffected by an IdP outage; interactive SSO sign-in must wait for the IdP to recover.
3. If key rotation broke auth, rollback the secret:
   kubectl rollout undo secret/kaireon-auth-config
   kubectl rollout restart deploy/kaireon-api
   
4. If clock skew, sync NTP:
   # On affected nodes
   sudo systemctl restart chronyd
   
5. If credential compromise is suspected:
   • Rotate all API keys and service account tokens immediately.
   • Revoke all active sessions.
   • Enable enhanced logging.
   • Notify the security team.

​

Prevention

• Implement rate limiting on auth endpoints (10 failures per IP per minute).
• Enable account lockout after 5 consecutive failures (30-minute window).
• Use short-lived tokens (15 minutes) with refresh token rotation.
• Monitor for credential stuffing patterns (many IPs, same usernames).
• Run quarterly penetration tests on auth flows.
• Sync NTP on all nodes; alert if clock drift >1 second.

​

Appendix: Useful Commands

​

Quick Health Check

# All-in-one health check
kubectl get pods -l app.kubernetes.io/part-of=kaireon | grep -v Running
kubectl top pods -l app.kubernetes.io/part-of=kaireon
kubectl exec -it deploy/kaireon-api -- curl -s localhost:3000/api/health | jq .
kubectl exec -it deploy/kaireon-redis -- redis-cli PING

​

Log Aggregation

# Tail all KaireonAI logs
kubectl logs -l app.kubernetes.io/part-of=kaireon --since=5m -f --max-log-requests=10

# Search for errors across all pods
kubectl logs -l app.kubernetes.io/part-of=kaireon --since=30m | grep -i error | sort | uniq -c | sort -rn

​

Metrics Quick Reference

# Decision throughput
sum(rate(kaireon_decisions_total[5m]))

# Error budget burn rate
1 - (sum(rate(http_responses_total{status!~"5.."}[1h])) / sum(rate(http_responses_total[1h])))

# P99 latency
histogram_quantile(0.99, sum(rate(kaireon_decision_latency_ms_bucket[5m])) by (le))