Flow Streaming Runtime

​

What FLOW_STREAMING_ENABLED gates today

​

Checkpoint persistence (lands the moment any broker connects)

import { writeCheckpoint, readCheckpoints } from "@/lib/flow/runtime/streaming/checkpoint";

// After every successful batch
await writeCheckpoint({
  prisma, runId, tenantId,
  checkpoint: { pipelineId, partition: 0, offset: 1234 },
});

// On consumer restart
const checkpoints = await readCheckpoints({ prisma, runId, tenantId });

​

Self-hoster setup

1. Set FLOW_STREAMING_ENABLED=true in the API + worker env. On its own this does not add any broker source consumer — it only flips the gate future consumers will check.
2. Add a streaming source consumer — because the consumer node kind is not in the source-node IR enum yet, this means extending lib/flow/ir/nodes/source.ts and adding an executor, then installing kafkajs (or @aws-sdk/client-kinesis / pulsar-client) and implementing the broker-specific consumer in kaireon-worker.
3. Restart the worker process — the existing kaireon-worker ECR repo is unwired in playground (no App Runner service); self-hosters point their orchestrator at it.
4. Each consumer run reads streamingCheckpoints on startup, seeks the broker to the saved offset, processes, and writeCheckpoints after each successful batch.

​

Honest limits carried into Phase 6.6 hardening

• Per-partition lag metrics on the per-node metrics dashboard
• Backpressure semantics beyond at-least-once
• UI hide for streaming node kinds when FLOW_STREAMING_ENABLED=false — the Visual canvas + Schedule tab don’t yet hide them with a “self-host required” CTA

​

Spec reference