Skip to main content
The Flow editor is the single surface for designing, inspecting, and operating pipelines. It lives at /data/flow-pipelines/[id]/edit. The legacy single-pane pipeline-flow-editor (PipelineNode/Edge model) was deleted on 2026-04-28 — every pipeline is now IR-native (irVersion: "1.0").

​
Layout

The AI Panel (Pipeline mode) is not in the right pane — it is a shell-level panel docked at the top-right of the page that auto-activates whenever the URL is under /data/flow-pipelines. A top bar shows the pipeline name, status pill, IR version pill, Publish button, and a Run-now button. A bottom strip shows the last run, next scheduled fire, DLQ count, and quick links. Run history lives on the dedicated Pipeline Runs page reached from the sidebar — the editor’s previous “Recent runs” left pane was redundant and was removed on 2026-05-02.

​
Tab reference

​
Visual

React Flow canvas, kind-coloured nodes (source / transform / validate / target / branch / join / enrich / outcomes), deterministic vertical waterfall layout. Each node card shows a one-line config summary inline (source: path · pattern · format; target: schema · loadMode; transform: op types; validate: rule count; outcomes: resolution mode · attribution window). Click a node → right pane swaps to Node Config, an editable per-kind form (the join kind is the one exception — it shows a read-only JSON view with a jump to the JSON IR tab). Above the canvas there is an Add Node toolbar with six buttons: **+ Transform / + Validate / + Enrich / + Branch / + Target /
  • Outcomes** (the addable kinds, from ADDABLE_KINDS in node-defaults.ts). Click a chainable button (Transform / Validate / Enrich / Branch) and kaireon inserts a default-configured node into the IR between the last upstream and your first sink, repoints the sinks that consumed the prior upstream so they now flow through the new node, and selects it so you can edit. + Target and + Outcomes are terminal sinks — they append a new node that consumes the same upstream as the existing chain rather than being spliced into it (see the Target and Outcomes sections below). Defaults intentionally use placeholder values (e.g. a transform with a rename: from_field → to_field op, or an outcomes node mapping the _kaireon_rid token) so you can tell at a glance which fields to customize.
Source and Join are not in the toolbar — every starter pipeline already has one source, multi-source pipelines are advanced, and Join requires picking two upstream node refs. Add those via the JSON IR tab or the AI panel. File-level archival is no longer a separate node kind — the source node’s atomicity config (successFolder / failureFolder) moves the source file to a date-templated archive folder on parse success and to a failure folder on parse error. See file ingestion for the full atomic staging flow.

​
JSON IR

Read/write IR editor with format-on-blur. Server-side validation goes through parsePipelineIR — invalid IR is rejected with the structured errors array rendered verbatim. Save creates a new IR version (POST /api/v1/pipelines/:id/ir) and bumps the version pill.

​
SQL Preview

Renders the exact INSERT … SELECT each target node would execute, using the same builders the runtime uses (buildLoadSql + buildSelectExpr). blue_green, incremental_watermark, and cdc_mirror targets render an explicit “preview not available — runtime load-mode helper handles it” note rather than fabricated SQL. GET /api/v1/pipelines/:id/sql-preview powers this tab.

​
Lineage

See Flow Lineage.

​
Schedule

See Flow Schedule.

​
Run history

Run history is a dedicated page at Sidebar → Data → Pipeline Runs (/data/flow-runs) — the editor’s previous in-pane “Recent runs” preview was redundant once the standalone page landed and was removed on 2026-05-02. It is a run-level table (Run id / Status / Started / Completed / Rows In / Rows Out / Processed / Error), one row per run.

​
Node config — what each form does

The right pane swaps to a per-kind form when you click a node:
  • Source — Path / file mask / format / wait policy. If the parent connector already supplies the bucket (S3 / GCS / Azure), the form hides the bucket input and only asks for the prefix.
  • Transform — Add op rows in order; each op has its own subform (cast type, formula, rename pair, etc.). A collapsible Sample row preview at the bottom takes one JSON row and shows the before / after diff for each op (added / removed / changed fields highlighted). Complex ops (aggregate, lookup_join, vector_embed, geo_resolve, sentiment_score, language_detect) preview as “not previewable client-side — run the pipeline” because they need runtime context. Note that summarize, vector_embed, geo_resolve, and sentiment_score in llm mode are passthrough at runtime (they return rows unchanged until a provider is wired); aggregate, lookup_join, sentiment_score in rule mode, and language_detect do run. See Transforms.
  • Validate — Row-level rules (notNull, regex, range, fieldType, maxLength), optional dataset-level row-count check, optional quarantine table for failed rows. The same Sample row preview at the bottom shows pass/fail per rule.
  • Enrich (Beta — not yet runnable) — Provider picker (llm_tag / geocode / ml_score) with a per-provider config block. The output-field input checks the destination DataSchema in real time: if the column doesn’t exist, it shows a red “missing column” badge with an inline + Add as <dataType> button that POSTs to /api/v1/schemas/fields with a sensible default type (string for llm_tag, float for geocode lat/lon and ml_score). The form itself carries a BETA banner because the enrich executor throws at runtime — the providers are not wired in this build. The node validates and saves, but a pipeline that contains it will fail when run; remove the enrich node (or implement the provider call) before running.
  • Branch — when predicate + then route per case, plus a default route. The route inputs are dropdowns populated from the current IR’s other nodes (excluding the branch itself and source nodes). Stale references (a then value that no longer matches any node id) render with a red border so they’re easy to spot and fix.
  • Target — Substantial form. Includes:
    • Reads from Select listing every other node in the IR — so multi-target IRs (one source fan-out to multiple destinations) are editable in the visual surface.
    • Schema name (schema.table, defaults to public.).
    • Load mode (append / truncate / upsert / blue_green / incremental_watermark / cdc_mirror). cdc_mirror applies an op-column change feed — configure the op column and CDC key. See Loading Modes & Validation.
    • Full-refresh hard warning — picking truncate on a one-source-one-target shape surfaces a recommend-blue_green banner with a one-click switch.
    • failOnEmptySource Switch (default on) for destructive modes — aborts before TRUNCATE / blue_green when upstream is empty.
    • backupBeforeLoad Switch for destructive modes — opt-in pre-load snapshot to <table>_backup_<runId> with retain-N pruning.
    • Mode-specific config blocks: upsertKey, watermarkColumn, and for cdc_mirror the op column (cdcOpColumn) + CDC key (upsertKey).
    • Inbound-FK probe when blue_green is picked — fires GET /api/v1/schemas/:id/incoming-fks and surfaces an amber warning listing referencing tables that would block the swap’s final DROP.
    • Inline mode-switch validation — empty upsertKey, missing watermark column, all-keys upsert, etc. all surface as inline errors that block save.
  • Outcomes — Inbound response-file ingestion (opens / clicks / bounces / conversions / unsubscribes) into interaction history and model learning; there is no output table. Includes:
    • Reads from Select — the upstream node carrying the vendor rows (sources / transforms, not other sinks).
    • Column mapping — a Resolve the offer by switch (Creative token _kaireon_rid or Offer id/name) that shows exactly one offer-resolution input, plus required customerId + outcomeKey and optional channelId / timestamp / conversionValue / eventId. Cleared optional columns store undefined (not "") so the IR’s .min(1) optionals don’t reject on save. A destructive inline error appears if neither offer-resolution column is set (mirrors the IR superRefine).
    • Attribution window (days) — number input, 1–365, default 7.
    • Suppression rules — an enable Switch (default on), the trigger outcomeKeys, a mechanism select (consent_revoke / do_not_contact), and — for consent_revoke — the revokePurposes list. The column mapping references the upstream vendor file columns, so these inputs deliberately do not autocomplete from the destination schema (an outcomes node has no destination schema). See Pipeline IR → Outcomes node for the full behaviour.
The toolbar above the canvas has + Target and + Outcomes alongside + Transform / + Validate / + Enrich / + Branch. Both sink buttons append a new node consuming the same upstream as the existing chain, without re-pointing it — pick a destination schema (target) or map the response columns (outcomes) in the right pane.

​
Honest residuals

Documented gaps, deferred to follow-up plans:
  • Drag-to-create-edge / right-click “Delete” on the visual canvas.
  • Pane collapse buttons (react-resizable-panels v4 typing settles).
  • Run-over-run trend chart (hardening pass).
  • Visual DAG highlight when clicking a lineage row (Wave 4 polish).
  • Selective replay-from-DLQ runtime (replayDlqOnly flag).
  • Sample row preview’s expression op shows the formula text rather than evaluating client-side — the server-side formula engine is the source of truth and replicating it in the browser would risk drift.
  • The 1500-line node-config-panel.tsx is partially extracted (constants + field-input primitives moved to components/flow-editor/forms/); the per-kind subforms still live in the panel. Full per-form file split is a follow-up refactor.
These aren’t broken — they aren’t built yet, and the UI calls them out by name where they would otherwise appear.