> ## Documentation Index > Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt > Use this file to discover all available pages before exploring further. # Worker Mode + Cron Drain > Operational runbook for the always-on-vs-cron-driven worker decision, Upstash quota math, cron-job.org setup, token rotation, and the deploy flow. > **Audience:** On-call engineers, SREs, platform operators > **First written:** 2026-05-06 (after a free-tier Upstash quota exhaustion incident) > **Related:** [`/api/v1/cron/drain-queues` reference](/api-reference/cron-drain-queues), [Env vars](/self-host/configure/env-vars), [Incident response](/self-host/runbooks/incident-response) This runbook captures everything you need to operate KaireonAI's BullMQ worker queues on free-tier or low-budget Redis. It exists because, on 2026-05-06, the playground hit Upstash's 500K-commands/month limit purely from idle worker polling — zero queued jobs, \~1.3M ops/month wasted on **BRPOPLPUSH** polls. The fix took \~30 minutes; the patterns below prevent it from happening again. *** ## 1. The two worker modes KaireonAI ships with two execution modes: | Mode | Toggle | Behavior | Job latency | Idle Redis cost | | --------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ------------------------ | | **Always-on** | `WORKER_INPROCESS=1` (default) | Five BullMQ workers run continuously inside the API container, each long-polling Redis with the **BRPOPLPUSH** command. | ≈ 0 (jobs picked up the moment they're enqueued) | \~30 ops/min permanently | | **Cron-driven drain** | `WORKER_INPROCESS=0` | No always-on workers. A scheduled cron hits `POST /api/v1/cron/drain-queues` every few minutes; each invocation connects, processes available jobs, disconnects. | ≤ cron interval (5 min default) | \~20 ops per invocation | For the queue mix on a typical deployment (`batch-jobs`, `dsar-jobs`, `journey-jobs`, `retrain-jobs`, `seed-jobs`), a 5-min cron sustains low-latency batch + DSAR + retrain workloads while burning \~170K ops/month idle — comfortably under Upstash free tier 500K. A 30-min cron drops to \~29K ops/month idle but adds up to 30 min of latency for batch / DSAR / retrain (still fine — these aren't real-time). ### When to pick which | Workload | Recommended | | ----------------------------------------------- | ------------------------------------------------------------------ | | Free-tier Redis (Upstash 500K, Fly Redis, etc.) | **`WORKER_INPROCESS=0`** + 5-min cron | | Paid Redis + active batch/journey traffic | `WORKER_INPROCESS=1` | | Mixed (paid Redis, but workers run elsewhere) | `WORKER_INPROCESS=0` on API + dedicated `kaireon-worker` container | *** ## 2. Upstash quota math (at a glance) Free-tier Upstash gives **500,000 Redis commands per month**. Here's where the budget goes: | Source | Ops / month | | ------------------------------------------------------ | ---------------- | | 5 BullMQ workers polling idle (`WORKER_INPROCESS=1`) | \~1,300,000 | | `WORKER_INPROCESS=0` + 5-min drain cron, idle queues | \~173,000 | | `WORKER_INPROCESS=0` + 15-min drain cron, idle queues | \~58,000 | | `WORKER_INPROCESS=0` + 30-min drain cron, idle queues | \~29,000 | | Per `/recommend` call with rate-limit + flow cache hit | \~5–8 | | Real /recommend traffic (e.g., 500 calls/day) | \~75,000–120,000 | **Rule of thumb:** with `WORKER_INPROCESS=0` + 5-min cron, you can sustain \~50K /recommend calls/month on free-tier Upstash before any rate-limit/cache cost becomes the binding factor. Above that, upgrade Redis. *** ## 3. Setup checklist (new deployment) If you're standing up a new App Runner service or migrating an existing one to cron-driven mode: ### 3.1 — Set the env vars on App Runner ```bash theme={null} ARN="arn:aws:apprunner:us-east-1:422500312304:service/kaireon-playground/" # 1. Snapshot the current SourceConfiguration so we don't drop unrelated env vars. aws apprunner describe-service --service-arn "$ARN" --region us-east-1 \ --query 'Service.SourceConfiguration' --output json > /tmp/sc.json # 2. Inject the new env vars (preserve existing ones). node -e " const fs = require('fs'); const sc = JSON.parse(fs.readFileSync('/tmp/sc.json')); const cur = sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables || {}; cur.WORKER_INPROCESS = '0'; cur.DRAIN_QUEUES_TOKEN = require('crypto').randomBytes(32).toString('hex'); cur.DRAIN_QUEUES_RATE_LIMIT = '12'; sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables = cur; fs.writeFileSync('/tmp/sc-new.json', JSON.stringify({SourceConfiguration: sc}, null, 2)); console.log('DRAIN_QUEUES_TOKEN=' + cur.DRAIN_QUEUES_TOKEN); " # 3. Apply. App Runner restarts the container with the new vars. aws apprunner update-service --service-arn "$ARN" --region us-east-1 \ --cli-input-json file:///tmp/sc-new.json \ --query 'OperationId' --output text ``` > **Why the snapshot-merge dance**: `update-service` replaces the *entire* runtime-environment-variables object on the service — passing only the new vars would silently delete `DATABASE_URL`, `REDIS_URL`, all your secrets. Always read-modify-write. ### 3.2 — Schedule the drain endpoint > **You may already have in-process drain.** When `CRON_SECRET` (or `CRON_TOKEN`) > is set, the in-process **maintenance scheduler** runs by default and > self-invokes `POST /api/v1/cron/drain-queues` every **15 minutes** (alongside > the other `/api/**/cron/*` jobs). On a single-container deployment that is > often enough on its own. An **external** scheduler (below) is only needed if > you have disabled the maintenance scheduler (`MAINTENANCE_SCHEDULER_ENABLED=false`), > run multiple API replicas, or want a tighter cadence than 15 minutes. Verify > which is firing with the log check in §3.3. Pick one scheduler. All work; pick by ergonomics + cost. #### Option A — cron-job.org (recommended, free, zero AWS resources) 1. Sign in at [https://cron-job.org](https://cron-job.org). 2. Create cronjob: * **Title:** `Kaireon drain queues` * **URL:** `https:///api/v1/cron/drain-queues` * **Schedule:** `Every 5 minutes` * **Save responses:** ✅ on 3. Advanced tab: * **Method:** POST * **Headers:** `X-Cron-Token: ` * **Timeout:** `60` seconds First execution fires within 5 min. Verify in cron-job.org's "Execution history" tab that you see `200 OK`. #### Option B — AWS EventBridge (cleanest if all-in on AWS) ```bash theme={null} # EventBridge → Lambda relay → App Runner (because EventBridge can't POST directly to App Runner URLs). # Or use EventBridge API Destinations if your App Runner service is behind API Gateway. ``` Cost: \~\$0.002/month at 5-min cadence (well under EventBridge's 14M-invocation free tier). #### Option C — GitHub Actions cron (simplest if your repo is public) `.github/workflows/drain-queues.yml`: ```yaml theme={null} name: Drain Queues on: schedule: - cron: "*/5 * * * *" workflow_dispatch: jobs: drain: runs-on: ubuntu-latest steps: - run: | curl -fsS -X POST "${{ secrets.PLAYGROUND_URL }}/api/v1/cron/drain-queues" \ -H "X-Cron-Token: ${{ secrets.DRAIN_QUEUES_TOKEN }}" ``` > Cost note: free for **public** repos. Private repos incur \~$53/month at 5-min cadence (each tick = 1 minute rounded × $0.008/min × 8,640 ticks/month minus 2,000 free min). Use cron-job.org or EventBridge for private repos. #### Option D — UptimeRobot or other uptime-monitor Same shape as Option A. Most uptime monitors support custom HTTP headers on free tiers. ### 3.3 — Verify ```bash theme={null} # 1. Endpoint healthy? curl -fsS -X POST "https:///api/v1/cron/drain-queues" \ -H "X-Cron-Token: $DRAIN_QUEUES_TOKEN" | jq # Expect: { "ok": true, "totalProcessed": 0, "queues": { ... all idleAt: 0 ... } } # 2. Cron actually firing? LG="/aws/apprunner/kaireon-playground//application" START=$(($(date +%s) * 1000 - 1200000)) # 20 min ago aws logs filter-log-events --region us-east-1 --log-group-name "$LG" \ --start-time $START --filter-pattern '"drain-queues complete"' \ --query 'length(events)' # Expect: ≥ 4 events in the last 20 min for a 5-min cadence. ``` *** ## 4. Token rotation procedure Periodic rotation reduces the blast radius of a leaked token. The drain endpoint accepts (in priority order): `DRAIN_QUEUES_TOKEN` → `CRON_SECRET` → `CRON_TOKEN`. The fallback chain lets you rotate without downtime. ```bash theme={null} # 1. Generate the new token. NEW_TOKEN=$(openssl rand -hex 32) echo "New: $NEW_TOKEN" # 2. Append it as the active DRAIN_QUEUES_TOKEN; keep the old token as # CRON_TOKEN (a fallback the endpoint still accepts) for the rotation window. ARN="arn:aws:apprunner:us-east-1:422500312304:service/kaireon-playground/" aws apprunner describe-service --service-arn "$ARN" --region us-east-1 \ --query 'Service.SourceConfiguration' --output json > /tmp/sc.json node -e " const fs = require('fs'); const sc = JSON.parse(fs.readFileSync('/tmp/sc.json')); const cur = sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables || {}; cur.CRON_TOKEN = cur.DRAIN_QUEUES_TOKEN; // demote old to fallback cur.DRAIN_QUEUES_TOKEN = '$NEW_TOKEN'; // new active sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables = cur; fs.writeFileSync('/tmp/sc-new.json', JSON.stringify({SourceConfiguration: sc}, null, 2)); " aws apprunner update-service --service-arn "$ARN" --region us-east-1 \ --cli-input-json file:///tmp/sc-new.json --query 'OperationId' --output text # 3. Wait for restart (~5 min), then update the cron-job.org header # to use $NEW_TOKEN instead of the old one. Test one tick manually. # 4. After the next cron-job.org tick succeeds with the new token, # remove CRON_TOKEN entirely: node -e " const fs = require('fs'); const sc = JSON.parse(fs.readFileSync('/tmp/sc.json')); const cur = sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables || {}; delete cur.CRON_TOKEN; sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables = cur; fs.writeFileSync('/tmp/sc-final.json', JSON.stringify({SourceConfiguration: sc}, null, 2)); " aws apprunner update-service --service-arn "$ARN" --region us-east-1 \ --cli-input-json file:///tmp/sc-final.json --query 'OperationId' --output text ``` The fallback chain means there's never a window where the cron is rejected by the endpoint. *** ## 5. Token blast radius (reference) What an attacker with `DRAIN_QUEUES_TOKEN` can do: | Allowed | Blocked | | --------------------------------------------------------------------------------- | --------------------------------------------------------- | | Hit `/api/v1/cron/drain-queues` repeatedly | All other `/api/v1/cron/*` endpoints (need `CRON_SECRET`) | | Trigger queue processing for already-enqueued jobs | Enqueue new jobs | | Cause cost amplification (compute via repeated drains, rate-limited at 12/min/IP) | Read tenant data, /recommend, customer entities | | | Modify roles, tenants, API keys | | | Touch AWS / ECR / RDS / App Runner config | | | Drop tables, delete the app | Realistic worst case: cost amplification on compute. Rate-limit blocks abuse beyond 12 req/min per IP. Even with a leaked token, no user data is exposed and the app cannot be destroyed. If a leak is suspected: rotate (above) and the leaked token becomes invalid on the next App Runner restart (\~5 min). *** ## 6. Common operations ### 6.1 — Switch between worker modes ```bash theme={null} # Always-on → cron-driven aws apprunner describe-service --service-arn "$ARN" --region us-east-1 \ --query 'Service.SourceConfiguration' --output json > /tmp/sc.json node -e " const fs = require('fs'); const sc = JSON.parse(fs.readFileSync('/tmp/sc.json')); sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables.WORKER_INPROCESS = '0'; fs.writeFileSync('/tmp/sc-new.json', JSON.stringify({SourceConfiguration: sc}, null, 2)); " aws apprunner update-service --service-arn "$ARN" --region us-east-1 \ --cli-input-json file:///tmp/sc-new.json ``` Reverse the value to go cron-driven → always-on. If you run a **dedicated `kaireon-worker` container** (the standalone `src/worker/index.ts` entrypoint, which always runs the five workers regardless of its own env), set `WORKER_INPROCESS=0` on the **API** container so the two don't both consume the same queues (double-counted metrics + job-ordering risk). Only run the API with `WORKER_INPROCESS=1` when there is **no** separate worker container. ### 6.2 — Trigger a one-shot drain manually ```bash theme={null} curl -fsS -X POST "https:///api/v1/cron/drain-queues?maxDurationMs=60000" \ -H "X-Cron-Token: $DRAIN_QUEUES_TOKEN" | jq ``` `maxDurationMs` caps wall-clock at 1 min (default) up to 10 min (hard cap). `maxConcurrentQueues=2` (default) means at most 2 BullMQ workers run inside one invocation. ### 6.3 — Move to a new Upstash database ```bash theme={null} # 1. Provision the new database in Upstash console; grab the standard # Redis URL (rediss://default:@:6379), NOT the REST URL. NEW_REDIS_URL='rediss://default:...@.upstash.io:6379' # 2. Smoke-test the URL first — bake into a local probe so we don't # discover a typo only after App Runner restarts. node -e " const Redis = require('ioredis'); const r = new Redis('$NEW_REDIS_URL', { lazyConnect: true, connectTimeout: 5000, maxRetriesPerRequest: 1 }); (async () => { await r.connect(); console.log('PING:', await r.ping()); await r.quit(); })(); " # Expect: PING: PONG # 3. Update App Runner. aws apprunner describe-service --service-arn "$ARN" --region us-east-1 \ --query 'Service.SourceConfiguration' --output json > /tmp/sc.json node -e " const fs = require('fs'); const sc = JSON.parse(fs.readFileSync('/tmp/sc.json')); sc.ImageRepository.ImageConfiguration.RuntimeEnvironmentVariables.REDIS_URL = '$NEW_REDIS_URL'; fs.writeFileSync('/tmp/sc-new.json', JSON.stringify({SourceConfiguration: sc}, null, 2)); " aws apprunner update-service --service-arn "$ARN" --region us-east-1 \ --cli-input-json file:///tmp/sc-new.json --query 'OperationId' --output text ``` > **Caveat**: switching Redis instances abandons any in-flight queued jobs in the old database. For playground / dev environments this is fine; for production, wait for queues to drain (`drain-queues` returns `totalProcessed: 0` for several consecutive ticks) before switching. *** ## 7. Deploy flow (this app) Standard deploy path for code or env-var changes: | Trigger | Command | What it does | | ------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------- | | Code change | `bash tools/scripts/build-and-deploy.sh` from repo root | Build Docker → push to ECR → trigger App Runner deployment | | Env-var change only | `aws apprunner update-service --cli-input-json file:///tmp/sc-new.json` | Restart container with new env vars (no image rebuild) | | Both | Run the deploy script first, then env-var update | Image rolls out, then env vars apply on the new image | **Always read-modify-write env vars** via the snapshot pattern above — `update-service` replaces the entire runtime-environment-variables block on the service. Each deploy takes \~4-5 minutes end-to-end: 1. Docker build (\~2 min) 2. ECR push (\~30 sec for incremental layers) 3. App Runner roll-out (\~2 min — container restart + health check) Verify with: ```bash theme={null} aws apprunner list-operations --service-arn "$ARN" --region us-east-1 \ --max-results 1 --query 'OperationSummaryList[0].[Id,Status,EndedAt]' --output text # Expect: SUCCEEDED curl -sm 8 https:///api/health | jq .uptime # Expect: uptime < 300 (i.e., container restarted within last 5 min) ``` *** ## 8. Incident: "Upstash quota exhausted" Symptoms: every Redis-backed feature returns `ERR max requests limit exceeded` (rate limit, cache, BullMQ enqueue). Email from Upstash announcing free-tier limit reached. ### Diagnosis ```bash theme={null} # Confirm via Upstash REST API: curl -sm 8 -H "Authorization: Bearer " \ "https://.upstash.io/PING" # If quota exhausted: { "error": "ERR max requests limit exceeded. Limit: 500000, Usage: 500000" } # Check current /recommend traffic to size the burn: psql "$DATABASE_URL" -c "SELECT COUNT(*) FROM decision_traces WHERE \"createdAt\" > NOW() - INTERVAL '30 days';" ``` If real /recommend traffic × \~5-8 ops/call \< 500K, the burn is from idle worker polling — apply the fix in §3. ### Resolution paths | Option | Effort | Cost | | -------------------------------------------------------------- | --------- | ----------------- | | `WORKER_INPROCESS=0` + cron drain (§3) | 30 min | \$0 | | New Upstash + same fix (cleaner reset) | 45 min | \$0 | | Upgrade Upstash to pay-as-you-go | 1 min | \~\$0.20/100K ops | | Move to a different Redis (Fly free, ElastiCache, self-hosted) | 1-2 hours | varies | The first option is recommended for free-tier deployments; it removes the burn at its source. The second is the "panic-restart" version when you also want a clean Redis with no abandoned queue state. *** ## 9. Dependency-vulnerability triage process When GitHub Dependabot opens alerts on the repo: ```bash theme={null} # 1. List open alerts. gh api repos///dependabot/alerts --paginate \ -q '.[] | select(.state=="open") | "\(.security_advisory.severity) [\(.dependency.package.ecosystem)] \(.dependency.package.name) — \(.security_advisory.summary) (alert #\(.number))"' # 2. For each, check the safe version + ancestry. npm view version npm ls # 3. Add an override in package.json (preferred when the vuln is transitive # and the parent hasn't released a fix yet): # "overrides": { "": "^" } # OR bump the direct dep if you control it. # 4. Reinstall + re-audit. npm install npm audit ``` The pattern shipped on 2026-05-06 closed 7 Dependabot alerts (4 high `@xmldom/xmldom`, 2 medium `postcss`/`fast-xml-parser`, 1 low `@tootallnate/once`) plus a bonus axios HIGH via npm overrides — see commit `86eff3f` for the exact change.