Skip to main content

​
Overview

The Data module is the foundation of Kaireon. It lets you connect to external data sources, define entity schemas that create real database tables, and build visual pipelines to transform and load data.

​
Connectors

Connectors define how Kaireon reaches your external data. Over 20 connector types are supported:
CategoryTypes
Cloud StorageAmazon S3, Google Cloud Storage, Azure Blob
DatabasesPostgreSQL, MySQL, Oracle, SQL Server, MongoDB
Data WarehousesSnowflake, BigQuery, Redshift, Databricks
StreamingKafka, Amazon Kinesis, Azure Event Hubs
APIsREST API, GraphQL, Salesforce, HubSpot
FilesSFTP, Local File Upload
Each connector type has its own configuration form with connection fields, authentication, and a Test Connection button to verify connectivity before saving.

​
Schemas

Schemas define your entity structure — customers, accounts, transactions, etc. When you create a schema, Kaireon creates an actual PostgreSQL table with the fields you define. Supported field types: text, integer, decimal, boolean, date, timestamp, json. Schemas are referenced by:
  • Enrichment stages in decision flows (to load customer data at decision time)
  • Computed values (formulas that reference customer.* fields)
  • Pipelines (as target destinations)

​
Pipelines

Pipelines are visual ETL workflows built with a drag-and-drop flow editor. Each pipeline has:
  • Source nodes — Read from a connector
  • Transform nodes — 14 transform types including cast, expression, rename, filter, hash, mask PII, join, aggregate, sort, deduplicate, pivot, unpivot, lookup, and custom SQL
  • Target nodes — Write to a schema table or external destination

​
Execution Config

Pipelines can run in batch or streaming mode with configurable:
  • Batch size and parallelism
  • Partitioning strategy
  • Error handling (skip, fail, dead-letter queue)
  • Scheduling (cron expressions)