> ## Documentation Index > Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt > Use this file to discover all available pages before exploring further. # Flow Scheduler > Built-in pipeline scheduler — runs in-process every minute. No external cron required. /api/v1/cron/flow-scheduler-tick remains for opt-in external orchestration. The Flow Scheduler fires due pipelines from their `ir.schedule`. **It runs in-process by default** via a 60-second `setInterval` registered in `instrumentation.ts` on server startup, so kaireon ships complete out of the box — no external cron service required. The same dispatch logic is also exposed as the `/api/v1/cron/flow-scheduler-tick` HTTP endpoint for self-hosters who prefer their own orchestrator (k8s CronJob, EventBridge, Vercel Cron, etc.). Multi-replica safety comes from a pg advisory lock — only one replica or HTTP caller's tick runs at a time per minute, the rest no-op with `{skipped: true}`. ## Configuration | Env var | Default | Purpose | | ---------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CRON_SECRET` | *required* | Shared secret for both the in-process ticker and the HTTP endpoint. The internal scheduler refuses to start if unset (logged warning) | | `FLOW_INTERNAL_SCHEDULER_ENABLED` | `true` | Set to `"false"` to disable in-process ticking — useful when you bring your own external scheduler | | `FLOW_SCHEDULER_INTERVAL_MS` | `60000` | Tick interval in milliseconds. Minimum 1000 | | `FLOW_INTERNAL_SCHEDULER_BASE_URL` | — | **Not read by the flow scheduler.** Run dispatch is in-process (`runBatch`), so there is no HTTP fetch to point anywhere. The ticker stopped reading this in 2026-08-15; setting it has no effect on pipeline scheduling. It still routes the separate **maintenance** scheduler, which does make real HTTP cron calls | | **PORT** | `3000` | Server port used to default the in-process scheduler base URL | ## Manual tick from the editor The Schedule tab inside the pipeline editor has a **Tick scheduler now** button that hits the HTTP endpoint with the same-origin session cookie. Useful when you've just saved a schedule and want to fire it without waiting for the next minute boundary. ## Endpoint ```http theme={null} GET /api/v1/cron/flow-scheduler-tick Authorization: Bearer $CRON_SECRET ``` (Or pass `x-cron-secret: $CRON_SECRET` header.) **Auth:** Fails closed when `CRON_SECRET` env is unset (matches the contract of `/api/v1/cron/approvals-expire` and other cron routes). **Response:** ```json theme={null} { "tickAt": "2026-04-27T20:00:00.000Z", "totalPipelines": 47, "scheduled": 12, "due": 3, "dispatched": 3, "failed": 0, "results": [ { "pipelineId": "...", "tenantId": "...", "fired": true, "status": 202 } ], "durationMs": 412 } ``` ## Per-kind decision logic Implemented as a pure due-pipelines helper in the platform's flow scheduling module — given the current set of schedules and the tick time, it returns the IDs that fire on this tick. | Kind | "Due" condition | | ------------ | -------------------------------------------------------------------------- | | **cron** | cron-parser yields any occurrence in `(lastRunAt, now]` (most recent wins) | | **interval** | `now - lastRunAt >= minutes * 60_000` | | **rrule** | `RRule.between(lastRunAt, now)` returns ≥ 1 occurrence | **`lastRunAt = null` (pipeline never ran):** treated as 24h lookback to avoid stampeding old, never-run pipelines that were created days/weeks ago. **Bad schedules** (malformed cron, malformed rrule) silently skip the pipeline — `parsePipelineIR` would have rejected the IR save earlier anyway. ## How it fires **Default (in-process):** The scheduler is **self-firing**. A `setInterval` registered by Next.js' instrumentation hook on server boot calls the same dispatch path every 60 seconds via the platform's internal-ticker module. No external orchestrator required. App Runner and any persistent-container deployment Just Work. Either way, the run itself is dispatched **in-process** (`runBatch` + the default executor registry) — the HTTP endpoint only triggers the *tick*; it never re-dispatches the run over HTTP. (A pipeline-run-over-HTTP path would 401 at middleware, which doesn't recognize the cron secret.) **Optional (external):** If you set `FLOW_INTERNAL_SCHEDULER_ENABLED=false` (e.g. you're on a serverless runtime with no persistent process), wire any external scheduler to the HTTP endpoint: | Setup | Approach | | --------------------------- | -------------------------------------------------------------------------------------------------------- | | **Self-hosted (k8s)** | A Kubernetes cron job resource hitting `/api/v1/cron/flow-scheduler-tick` every minute | | **Self-hosted (Linux box)** | `* * * * * curl -fsSL -H "x-cron-secret: $CRON_SECRET" https://your-app/api/v1/cron/flow-scheduler-tick` | | **AWS** | EventBridge Schedule → API destination | | **Vercel** | A Vercel Cron Job (vercel.json `crons` array) hitting the endpoint | ## Idempotency * Calling the endpoint multiple times in one minute does not double-fire a pipeline — once a run is dispatched, `lastRunAt` advances to the fire time, and the next tick's `(lastRunAt, now]` window excludes the same occurrence. * Per-tick work is O(n) over all IR-native pipelines + 1 IR load (from the DB) each. Phase 6.6 hardening adds an index + a "schedule\_only" projection. ## File-arrival triggers (push + poll) File-arrival triggers complement schedules. Use them when a pipeline should fire **when a sentinel file shows up** instead of on a clock — the canonical "I dropped customers.csv into the inbox, go process it" pattern. ### IR shape ```json theme={null} { "trigger": { "kind": "file_arrival", "sourceId": "src", "controlFilePattern": { "type": "glob", "value": "_ready.json" }, "debounceSeconds": 60, "deadline": { "windowMinutes": 60, "onMiss": "alert" } } } ``` * **`sourceId`** — the source node in the same IR that will consume the file when the run fires. * **`controlFilePattern`** (optional) — the sentinel that *fires* the run. **Per-pipeline.** When omitted, the trigger uses the referenced source's own pattern (so the data file's arrival fires the run directly). See "Multiple pipelines, same folder" below for the multi-mask story. * **`debounceSeconds`** (optional, default 60) — minimum time between consecutive fires. Prevents a slow run + a new file mid-run from double-firing. * **`deadline.windowMinutes` / `onMiss`** (optional) — if no file arrives within `windowMinutes` of `lastRunAt`, take the `onMiss` action (`alert`, `fail`, or `skip`). ### Multiple pipelines, same folder `controlFilePattern` is per-pipeline. When several pipelines watch the same inbox, each declares its own sentinel mask and only fires when its specific sentinel arrives. Example: three pipelines sharing `s3://kaireon-e2e-tests/inbox/`: ```json theme={null} // load-customers "trigger": { "kind": "file_arrival", "sourceId": "src", "controlFilePattern": { "type": "glob", "value": "customers.done" } } // load-accounts "trigger": { "kind": "file_arrival", "sourceId": "src", "controlFilePattern": { "type": "glob", "value": "accounts.done" } } // load-propositions "trigger": { "kind": "file_arrival", "sourceId": "src", "controlFilePattern": { "type": "glob", "value": "propositions.done" } } ``` Drop `customers.done` → only `load-customers` fires. Drop both `customers.done` and `accounts.done` on the same tick → both fire, independently. The source executor reads each pipeline's own data-file pattern (e.g. `customers.csv`, `accounts.csv`), so the data files don't collide either. Make the masks **disjoint**. Two pipelines whose `controlFilePattern` both match the same key (e.g. both globs include `*.done`) means whichever EventBridge dispatch arrives first wins — the poll path is fine because each pipeline probes independently, but the push resolver needs a single match to identify the target pipeline. Use distinct prefixes / suffixes (`customers.*.done`, `accounts.*.done`) when in doubt. After a successful fire, the **sentinel is archived to `atomicity.successFolder`** (the same folder the source executor uses for data files), so the same trigger doesn't re-fire on every subsequent tick. A failed run moves the sentinel to `atomicity.failureFolder` instead. Drop a fresh sentinel to retry. ### How it fires (two paths) | Path | Latency | Setup | | ------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------- | | **Poll (default)** | up to one scheduler tick (≤ 60s) | Zero config — the in-process scheduler polls every active file\_arrival trigger on each tick | | **Push (optional)** | sub-second | Wire S3 → EventBridge → API destination (or any webhook source) to `POST /api/v1/triggers/file-arrival` | Both paths converge on the same dispatch entry — the run drawer will show `triggerSource: "file_arrival"`. ### Push endpoint ```http theme={null} POST /api/v1/triggers/file-arrival Authorization: Bearer $CRON_SECRET Content-Type: application/json ``` Native payload: ```json theme={null} { "pipelineId": "94640f58-9fcc-…", "tenantId": "4b148e40-…", "objectKey": "inbox/_ready.json" } ``` S3 EventBridge envelope (forwarded verbatim from an EventBridge target): ```json theme={null} { "detail": { "bucket": { "name": "kaireon-e2e-tests" }, "object": { "key": "inbox/_ready.json" } } } ``` When the EventBridge shape arrives, the endpoint scans all active pipelines for a matching `(bucket, controlFilePattern)` pair, probes the source path to confirm the key really matches, then dispatches the run. **Auth:** the `Authorization` (or `x-cron-secret`) header must equal `CRON_SECRET`. Per-pipeline trigger secrets are on the roadmap. **Idempotency:** dispatching the same file twice is safe — the source executor's `pendingFinalizers` machinery moves the file to `.archive/` or `.failed/` after the first run, so the second run finds no matching files and no-ops via `onMissAction`. ### Deadline enforcement When the scheduler polls and finds no matching file but the trigger carries a `deadline`, it checks whether the deadline window has elapsed since `lastRunAt` (or pipeline `createdAt` for never-run pipelines): | `onMiss` | Behavior | | -------- | --------------------------------------------------------------------------------------------- | | `alert` | Emits a system-health warning (visible in the alerts dashboard). | | `fail` | Same alert + writes a synthetic failed `PipelineRun` so health dashboards count the SLA miss. | | `skip` | Logs and moves on. Useful in development. | Each `(pipeline, anchorTime)` pair fires the action at most once — the dedupe key is the pipeline's `lastRunAt` (or `createdAt`), so a fresh fire that advances `lastRunAt` re-arms the deadline. ## Roadmap * **Per-tenant scheduler dashboard** showing miss / lag / failed dispatches. * **Per-pipeline trigger secrets** for tenant-isolated push webhooks (today the file-arrival endpoint validates against `CRON_SECRET`). * **SFTP fanotify / inotify-driven local\_fs triggers** so on-prem deployments get the same sub-second push latency S3 → EventBridge delivers.