ir.schedule. It
runs in-process by default via a 60-second setInterval registered
in instrumentation.ts on server startup, so kaireon ships complete
out of the box — no external cron service required. The same dispatch
logic is also exposed as the /api/v1/cron/flow-scheduler-tick HTTP
endpoint for self-hosters who prefer their own orchestrator (k8s
CronJob, EventBridge, Vercel Cron, etc.).
Multi-replica safety comes from a pg advisory lock — only one replica
or HTTP caller’s tick runs at a time per minute, the rest no-op with
{skipped: true}.
Configuration
Manual tick from the editor
The Schedule tab inside the pipeline editor has a Tick scheduler now button that hits the HTTP endpoint with the same-origin session cookie. Useful when you’ve just saved a schedule and want to fire it without waiting for the next minute boundary.Endpoint
x-cron-secret: $CRON_SECRET header.)
Auth: Fails closed when CRON_SECRET env is unset (matches the
contract of /api/v1/cron/approvals-expire and other cron routes).
Response:
Per-kind decision logic
Implemented as a pure due-pipelines helper in the platform’s flow scheduling module — given the current set of schedules and the tick time, it returns the IDs that fire on this tick.lastRunAt = null (pipeline never ran): treated as 24h lookback to
avoid stampeding old, never-run pipelines that were created days/weeks
ago.
Bad schedules (malformed cron, malformed rrule) silently skip the
pipeline — parsePipelineIR would have rejected the IR save earlier
anyway.
How it fires
Default (in-process): The scheduler is self-firing. AsetInterval registered by Next.js’ instrumentation hook on server
boot calls the same dispatch path every 60 seconds via the platform’s
internal-ticker module. No external orchestrator required. App
Runner and any persistent-container deployment Just Work.
Either way, the run itself is dispatched in-process (runBatch +
the default executor registry) — the HTTP endpoint only triggers the
tick; it never re-dispatches the run over HTTP. (A pipeline-run-over-HTTP
path would 401 at middleware, which doesn’t recognize the cron secret.)
Optional (external): If you set
FLOW_INTERNAL_SCHEDULER_ENABLED=false (e.g. you’re on a serverless
runtime with no persistent process), wire any external scheduler to the
HTTP endpoint:
Idempotency
- Calling the endpoint multiple times in one minute does not double-fire
a pipeline — once a run is dispatched,
lastRunAtadvances to the fire time, and the next tick’s(lastRunAt, now]window excludes the same occurrence. - Per-tick work is O(n) over all IR-native pipelines + 1 IR load (from the DB) each. Phase 6.6 hardening adds an index + a “schedule_only” projection.
File-arrival triggers (push + poll)
File-arrival triggers complement schedules. Use them when a pipeline should fire when a sentinel file shows up instead of on a clock — the canonical “I dropped customers.csv into the inbox, go process it” pattern.IR shape
sourceId— the source node in the same IR that will consume the file when the run fires.controlFilePattern(optional) — the sentinel that fires the run. Per-pipeline. When omitted, the trigger uses the referenced source’s own pattern (so the data file’s arrival fires the run directly). See “Multiple pipelines, same folder” below for the multi-mask story.debounceSeconds(optional, default 60) — minimum time between consecutive fires. Prevents a slow run + a new file mid-run from double-firing.deadline.windowMinutes/onMiss(optional) — if no file arrives withinwindowMinutesoflastRunAt, take theonMissaction (alert,fail, orskip).
Multiple pipelines, same folder
controlFilePattern is per-pipeline. When several pipelines watch the
same inbox, each declares its own sentinel mask and only fires when its
specific sentinel arrives. Example: three pipelines sharing
s3://kaireon-e2e-tests/inbox/:
customers.done → only load-customers fires. Drop both
customers.done and accounts.done on the same tick → both fire,
independently. The source executor reads each pipeline’s own data-file
pattern (e.g. customers.csv, accounts.csv), so the data files don’t
collide either.
Make the masks disjoint. Two pipelines whose controlFilePattern
both match the same key (e.g. both globs include *.done) means
whichever EventBridge dispatch arrives first wins — the poll path is
fine because each pipeline probes independently, but the push resolver
needs a single match to identify the target pipeline. Use distinct
prefixes / suffixes (customers.*.done, accounts.*.done) when in
doubt.
After a successful fire, the sentinel is archived to
atomicity.successFolder (the same folder the source executor uses
for data files), so the same trigger doesn’t re-fire on every
subsequent tick. A failed run moves the sentinel to
atomicity.failureFolder instead. Drop a fresh sentinel to retry.
How it fires (two paths)
Both paths converge on the same dispatch entry — the run drawer will
show
triggerSource: "file_arrival".
Push endpoint
(bucket, controlFilePattern) pair, probes
the source path to confirm the key really matches, then dispatches the
run.
Auth: the Authorization (or x-cron-secret) header must equal
CRON_SECRET. Per-pipeline trigger secrets are on the roadmap.
Idempotency: dispatching the same file twice is safe — the source
executor’s pendingFinalizers machinery moves the file to .archive/
or .failed/ after the first run, so the second run finds no matching
files and no-ops via onMissAction.
Deadline enforcement
When the scheduler polls and finds no matching file but the trigger carries adeadline, it checks whether the deadline window has elapsed
since lastRunAt (or pipeline createdAt for never-run pipelines):
Each
(pipeline, anchorTime) pair fires the action at most once — the
dedupe key is the pipeline’s lastRunAt (or createdAt), so a fresh
fire that advances lastRunAt re-arms the deadline.
Roadmap
- Per-tenant scheduler dashboard showing miss / lag / failed dispatches.
- Per-pipeline trigger secrets for tenant-isolated push webhooks
(today the file-arrival endpoint validates against
CRON_SECRET). - SFTP fanotify / inotify-driven local_fs triggers so on-prem deployments get the same sub-second push latency S3 → EventBridge delivers.