Drain Queues (Cron)

1. Always-on (WORKER_INPROCESS=1, the legacy default): five workers run continuously inside the API container, long-polling Redis with blocking pop operations. Job latency ≈ 0. Idle Redis cost ≈ 30 ops/min permanently.
2. Cron-driven drain (WORKER_INPROCESS=0, recommended for free-tier Redis): no always-on workers. A scheduled cron hits POST /api/v1/cron/drain-queues every few minutes. Each invocation connects, processes available jobs, and disconnects. Job latency ≤ cron interval. Idle Redis cost ≈ 20 ops per invocation × invocations/day.

​

When to use

​

Set the toggle

# Disables in-process workers — the API process no longer spins up a worker on boot.
WORKER_INPROCESS=0
# Required — used for cron auth
CRON_SECRET=<long-random-hex>

​

POST /api/v1/cron/drain-queues

​

Auth

• Sliding-window per-IP rate limit — DRAIN_QUEUES_RATE_LIMIT requests/minute (default 12). Returns 429 with Retry-After when exceeded.
• Optional IP allowlist — set CRON_ALLOWED_IPS=ip1,ip2,… (comma-separated). When set, only requests from listed IPs (matched against X-Forwarded-For left-most) pass. When unset, IP check is skipped (token-only).
• Constant-time token compare — prevents timing-attack token discovery.

​

Query parameters

​

Response 200

{
  "ok": true,
  "durationMs": 1234,
  "totalProcessed": 3,
  "totalFailed": 0,
  "queues": {
    "batch-jobs":   { "jobsProcessed": 1, "jobsFailed": 0, "idleAt": 412 },
    "dsar-jobs":    { "jobsProcessed": 0, "jobsFailed": 0, "idleAt": 0 },
    "journey-jobs": { "jobsProcessed": 2, "jobsFailed": 0, "idleAt": 891 },
    "retrain-jobs": { "jobsProcessed": 0, "jobsFailed": 0, "idleAt": 0 },
    "seed-jobs":    { "jobsProcessed": 0, "jobsFailed": 0, "idleAt": 0 }
  }
}

​

Error codes

​

Scheduling — pick one

​

Option A — GitHub Actions cron (simplest, free)

name: Drain Queues
on:
  schedule:
    - cron: "*/5 * * * *"   # every 5 minutes
  workflow_dispatch:
jobs:
  drain:
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -fsS -X POST "${{ secrets.PLAYGROUND_URL }}/api/v1/cron/drain-queues" \
            -H "X-Cron-Secret: ${{ secrets.CRON_SECRET }}" \
            -H "Content-Type: application/json"

​

Option B — AWS EventBridge schedule (preferred when already on AWS)

aws events put-rule --name kaireon-drain-queues \
  --schedule-expression "rate(5 minutes)" \
  --region us-east-1

aws events put-targets --rule kaireon-drain-queues \
  --targets '[{
    "Id": "1",
    "Arn": "arn:aws:apigateway:us-east-1:apigateway/.../api/v1/cron/drain-queues",
    "HttpParameters": {
      "HeaderParameters": { "X-Cron-Secret": "<CRON_SECRET>" }
    }
  }]'

​

Option C — External uptime monitor (cheap, hands-off)

• URL: https://playground.kaireonai.com/api/v1/cron/drain-queues
• Method: HTTP POST
• Headers: X-Cron-Secret: <CRON_SECRET>
• Schedule: */5 * * * *

​

Option D — Self-managed (k8s CronJob, supervised cron, etc.)

curl -fsS -X POST "$KAIREON_URL/api/v1/cron/drain-queues" \
  -H "X-Cron-Secret: $CRON_SECRET"

​

Cost math

ops_per_invocation = 5 queues × 4 ops (getJobCounts) = 20
invocations_per_month = 12 × 24 × 30 = 8,640  (every 5 min)
ops_per_month_idle = ~173,000

Free tier (Upstash) = 500,000 ops/month
Headroom = ~327,000 ops for actual jobs + /recommend rate-limit + flow cache

ops_per_month_idle = 20 × 48 × 30 = ~29,000
Headroom = ~471,000 ops for everything else

​

Caveats

• Job-failure semantics differ from always-on: in always-on mode, a failed job retries via BullMQ’s exponential backoff immediately. In cron-driven mode, retries are picked up on the next tick. For low-frequency workloads this is fine. For SLA-sensitive workloads, run always-on workers on paid Redis.
• Long-running jobs (>maxDurationMs) will be aborted mid-flight when the worker closes. They’ll be re-enqueued by BullMQ’s stalled-job detector on the next tick. Set maxDurationMs higher than your longest expected job, or split jobs into smaller chunks.
• The drain endpoint is idempotent — re-hitting it during an in-flight invocation just no-ops on jobs already in-flight (BullMQ’s lock semantics).