> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Drain Queues (Cron)

> Cron-driven BullMQ worker drain. Replaces always-on workers when running on Redis quotas where idle BLPOP polling is the binding cost.

KaireonAI's BullMQ workers have two execution modes:

1. **Always-on** (`WORKER_INPROCESS=1`, the legacy default): five workers run continuously inside the API container, long-polling Redis with blocking pop operations. Job latency ≈ 0. Idle Redis cost ≈ 30 ops/min permanently.
2. **Cron-driven drain** (`WORKER_INPROCESS=0`, recommended for free-tier Redis): no always-on workers. A scheduled cron hits `POST /api/v1/cron/drain-queues` every few minutes. Each invocation connects, processes available jobs, and disconnects. Job latency ≤ cron interval. Idle Redis cost ≈ 20 ops per invocation × invocations/day.

For a 5-minute cron with 5 idle queues, that's \~170K ops/month — comfortably under the Upstash free tier of 500K, vs \~1.3M/month for always-on workers with no actual jobs.

## When to use

| Workload                                                            | Recommended mode                                                         |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Free-tier Redis or low-traffic playground                           | **Cron-driven**, every 5 min                                             |
| Production with paid Redis + active batch/journey traffic           | Always-on                                                                |
| Mixed (paid Redis, but workers run elsewhere as a separate service) | Cron-driven on the API; standalone worker container for the heavy queues |

## Set the toggle

Set the env var on your API container and redeploy:

```bash theme={null}
# Disables in-process workers — the API process no longer spins up a worker on boot.
WORKER_INPROCESS=0
# Required — used for cron auth
CRON_SECRET=<long-random-hex>
```

When `WORKER_INPROCESS` is unset or `=1`, the API container runs the legacy always-on worker. When `=0`, only `/api/v1/cron/drain-queues` produces job consumption.

## POST /api/v1/cron/drain-queues

Drain queued jobs across the 5 BullMQ queues (`batch-jobs`, `dsar-jobs`, `journey-jobs`, `retrain-jobs`, `seed-jobs`).

### Auth

Three env vars are accepted in priority order:

| Env var              | Scope                          | When to use                                                                                           |
| -------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------- |
| `DRAIN_QUEUES_TOKEN` | this endpoint only             | **Recommended for external schedulers** (cron-job.org, etc.) — narrow blast radius if the token leaks |
| `CRON_SECRET`        | shared by all `/api/v1/cron/*` | Use when the same internal scheduler hits multiple cron endpoints                                     |
| `CRON_TOKEN`         | backwards-compat alias         | Pre-existing envs that haven't migrated                                                               |

Headers accepted: `X-Cron-Secret`, `X-Cron-Token`, or `Authorization: Bearer <token>`.

**Hardening (also active by default):**

* **Sliding-window per-IP rate limit** — `DRAIN_QUEUES_RATE_LIMIT` requests/minute (default `12`). Returns `429` with `Retry-After` when exceeded.
* **Optional IP allowlist** — set `CRON_ALLOWED_IPS=ip1,ip2,…` (comma-separated). When set, only requests from listed IPs (matched against `X-Forwarded-For` left-most) pass. When unset, IP check is skipped (token-only).
* **Constant-time token compare** — prevents timing-attack token discovery.

### Query parameters

| Parameter             | Type   | Default         | Description                                                                                                                                                                  |
| --------------------- | ------ | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `queue`               | string | (all 5)         | Drain only this queue. Allowed: `batch-jobs`, `dsar-jobs`, `journey-jobs`, `retrain-jobs`, `seed-jobs`.                                                                      |
| `maxDurationMs`       | number | `60000` (1 min) | Hard wall-clock cap. Max `600000` (10 min). The endpoint exits as soon as queues report idle, but never runs longer than this.                                               |
| `maxConcurrentQueues` | number | `2`             | How many queues run BullMQ workers in parallel inside one invocation. Caps concurrent Redis connections so idle ops stay low on free-tier Redis. Bump to 5 when self-hosted. |
| `maxJobsPerQueue`     | number | unbounded       | Safety stop per queue. Useful when chaining short cron ticks.                                                                                                                |

### Response `200`

```json theme={null}
{
  "ok": true,
  "durationMs": 1234,
  "totalProcessed": 3,
  "totalFailed": 0,
  "queues": {
    "batch-jobs":   { "jobsProcessed": 1, "jobsFailed": 0, "idleAt": 412 },
    "dsar-jobs":    { "jobsProcessed": 0, "jobsFailed": 0, "idleAt": 0 },
    "journey-jobs": { "jobsProcessed": 2, "jobsFailed": 0, "idleAt": 891 },
    "retrain-jobs": { "jobsProcessed": 0, "jobsFailed": 0, "idleAt": 0 },
    "seed-jobs":    { "jobsProcessed": 0, "jobsFailed": 0, "idleAt": 0 }
  }
}
```

`idleAt` is the wall-clock-ms-since-start when the queue first reported idle. `0` means the queue was already empty when probed (no worker was started — the cheap `getJobCounts` probe runs and the endpoint moves on).

### Error codes

| Code  | Reason                            |
| ----- | --------------------------------- |
| `400` | Unknown `queue` parameter.        |
| `401` | Missing or invalid `CRON_SECRET`. |
| `500` | `REDIS_URL` not configured.       |

## Scheduling — pick one

### Option A — GitHub Actions cron (simplest, free)

`.github/workflows/drain-queues.yml`:

```yaml theme={null}
name: Drain Queues
on:
  schedule:
    - cron: "*/5 * * * *"   # every 5 minutes
  workflow_dispatch:
jobs:
  drain:
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -fsS -X POST "${{ secrets.PLAYGROUND_URL }}/api/v1/cron/drain-queues" \
            -H "X-Cron-Secret: ${{ secrets.CRON_SECRET }}" \
            -H "Content-Type: application/json"
```

GitHub's free tier gives 2,000 minutes/month — plenty for 5-min cron.

### Option B — AWS EventBridge schedule (preferred when already on AWS)

```bash theme={null}
aws events put-rule --name kaireon-drain-queues \
  --schedule-expression "rate(5 minutes)" \
  --region us-east-1

aws events put-targets --rule kaireon-drain-queues \
  --targets '[{
    "Id": "1",
    "Arn": "arn:aws:apigateway:us-east-1:apigateway/.../api/v1/cron/drain-queues",
    "HttpParameters": {
      "HeaderParameters": { "X-Cron-Secret": "<CRON_SECRET>" }
    }
  }]'
```

(For App Runner, you may need an API Gateway connector or a Lambda intermediary since EventBridge can't directly POST to App Runner URLs.)

### Option C — External uptime monitor (cheap, hands-off)

Services like [Cron-Job.org](https://cron-job.org), [EasyCron](https://www.easycron.com), or [UptimeRobot](https://uptimerobot.com) can hit any HTTPS URL on a schedule. Configure:

* URL: `https://playground.kaireonai.com/api/v1/cron/drain-queues`
* Method: HTTP POST
* Headers: `X-Cron-Secret: <CRON_SECRET>`
* Schedule: `*/5 * * * *`

### Option D — Self-managed (k8s CronJob, supervised cron, etc.)

Use whatever scheduler your platform provides. Each tick should run:

```bash theme={null}
curl -fsS -X POST "$KAIREON_URL/api/v1/cron/drain-queues" \
  -H "X-Cron-Secret: $CRON_SECRET"
```

## Cost math

For a tenant with **zero queued jobs** (the common idle case on playground):

```
ops_per_invocation = 5 queues × 4 ops (getJobCounts) = 20
invocations_per_month = 12 × 24 × 30 = 8,640  (every 5 min)
ops_per_month_idle = ~173,000

Free tier (Upstash) = 500,000 ops/month
Headroom = ~327,000 ops for actual jobs + /recommend rate-limit + flow cache
```

For **every-30-minute** cron (lower latency tolerance):

```
ops_per_month_idle = 20 × 48 × 30 = ~29,000
Headroom = ~471,000 ops for everything else
```

In practice you can run a 5-min cron on a free-tier Redis with no concerns until you reach hundreds of /recommend calls per minute, at which point the rate-limiter cost dominates and you should upgrade Redis anyway.

## Caveats

* **Job-failure semantics differ from always-on**: in always-on mode, a failed job retries via BullMQ's exponential backoff *immediately*. In cron-driven mode, retries are picked up on the next tick. For low-frequency workloads this is fine. For SLA-sensitive workloads, run always-on workers on paid Redis.
* **Long-running jobs** (>`maxDurationMs`) will be aborted mid-flight when the worker closes. They'll be re-enqueued by BullMQ's stalled-job detector on the next tick. Set `maxDurationMs` higher than your longest expected job, or split jobs into smaller chunks.
* **The drain endpoint is idempotent** — re-hitting it during an in-flight invocation just no-ops on jobs already in-flight (BullMQ's lock semantics).