YAML Connectors & Plugin SDK

KaireonAI supports a three-tier connector model for wiring HTTP/REST sources beyond the built-in registry. All three tiers are wired end-to-end: server-side validation, REST + MCP registration, and an authoring page at Data → YAML Connectors (paste a spec, validate, register — or generate a draft with the AI generator).

Specs are validated, not yet runnable. The registry is in-memory, so a registered spec does not survive a server restart or redeploy, and registering creates no connector record. A YAML connector therefore never appears in the New Pipeline connector picker and cannot be used as a pipeline source — the IR source node accepts a closed set of file/object-storage kinds, which a YAML spec is not among. Use this tier to author and validate specs; ingestion for YAML connectors is not wired yet.

Three tiers

Tier	Mechanism	When
YAML spec	Declarative HTTP/REST connector	90% of SaaS APIs (Salesforce, Stripe, HubSpot, Klaviyo, …)
TypeScript plugin	Typed plugin SDK with a single `defineConnector` factory	Streaming + complex protocols (Kafka, Snowflake COPY, JDBC)
AI-generated YAML	AI drafts a spec from a docs URL, OpenAPI snippet, or plain-text hint	Net-new HTTP APIs you’d rather not hand-author

YAML spec format

kind: connector
version: "1.0"
id: salesforce_rest
displayName: Salesforce REST
category: crm
auth:
  - type: oauth2
    authorizeUrl: https://login.salesforce.com/services/oauth2/authorize
    tokenUrl: https://login.salesforce.com/services/oauth2/token
    scopes: [api, refresh_token]
    clientIdRef: "{{secrets.clientId}}"
    clientSecretRef: "{{secrets.clientSecret}}"
endpoints:
  - id: query
    method: GET
    url: "{{instance_url}}/services/data/v59.0/query"
    params:
      q: "{{soql}}"
    pagination:
      type: cursor
      cursorField: nextRecordsUrl
      pageSize: 200
    rateLimit:
      requestsPerSecond: 10
read:
  primary: query
  resourceTypes: [Account, Contact, Lead, Opportunity]

Validated by the connector-YAML parser — js-yaml load + Zod safeParse, same two-phase pattern as the pipeline-IR parser. Invalid specs are rejected before they enter the registry.

Auth types

none, api_key (header or query), basic, oauth2 (operator passes a pre-fetched accessToken in secrets.accessToken; Phase 5 doesn’t auto-refresh, that’s a follow-up).

Pagination types

none, cursor (follows a response field until absent), offset (numeric offset+limit), page (page number). Each stops on a short page or maxPages cap (default 100).

Rate limit

requestsPerSecond paces inter-page sleeps. burstSize is reserved for a future token-bucket implementation.

HTTP runtime

The YAML endpoint executor runs the call against the named spec, given the endpoint id, request parameters, secrets, and an abort signal:

Resolve auth → headers + query params
Substitute {{var}} templates against (params + secrets)
SSRF-validate the URL via lib/security/url-validator.validateAndResolve
Fetch with rate-limit pacing
Walk pagination
Extract rows via responseRowsPath (defaults to root)

Plugin SDK

import { z } from "zod";
import { defineConnector } from "@/lib/flow/connectors/plugin-sdk";

export default defineConnector({
  id: "kafka",
  displayName: "Apache Kafka",
  category: "streaming",
  configSchema: z.object({
    brokers: z.array(z.string()),
    topic: z.string(),
    consumerGroup: z.string(),
  }),
  read: async function* (ctx) {
    // implementation; yields { key, value, partition, offset, ... }
  },
  testConnection: async (config) => ({ ok: true }),
});

Plugins register themselves in the in-memory connectorRegistry via connectorRegistry.registerPlugin(plugin). Plugin imports must be explicit (no filesystem auto-discovery in Phase 5).

MCP `createYamlConnector`

Promoted from Phase 2b stub. Accepts a YAML text body, validates via parseConnectorYaml, registers on success.

{
  "ok": true,
  "id": "salesforce_rest",
  "displayName": "Salesforce REST"
}

Or:

{
  "ok": false,
  "errors": ["auth.0.tokenUrl: must be https://"]
}

REST endpoints

Two REST routes back the same in-memory registry. Both require an admin or editor role plus a tenant; the authoring page posts to them.

Route	Purpose	Response
`POST /api/v1/connectors/yaml`	Validate + register a spec — body `{ yaml }`	`201 { ok, id, displayName, category }`; `400 { ok: false, errors }` on invalid YAML; `409` on an id collision
`GET /api/v1/connectors/yaml`	List registered YAML + plugin connectors	`200 [{ id, displayName, category, kind }]`

A registered spec’s id is then available to Flow IR source nodes.

AI generator

POST /api/v1/ai/generate-yaml-connector drafts a spec for you. Body: { docsUrl?, openapi?, hint? } — at least one is required. The generator fetches the docs (SSRF-guarded), prompts the tenant’s configured model via generateObject against the connector-spec Zod schema, and retries up to twice on validation failure — re-prompting each time with the structured errors.

{
  "ok": true,
  "retries": 0,
  "tokensUsed": 1830,
  "spec": { "kind": "connector", "version": "1.0", "id": "brevo", "...": "..." },
  "yaml": "kind: connector\nversion: \"1.0\"\n..."
}

On failure it returns { ok: false, errors, retries, tokensUsed } (HTTP 200). The route requires an admin/editor role and the tenant’s Flow IR feature flag; the generator panel is collapsible on the Data → YAML Connectors page.

Not yet supported

Marketplace UI — a browse/install catalog for shared specs (distinct from the authoring page, which ships today)
Bulk migration of the built-in registry’s HTTP-shaped connectors to YAML specs — the 80 built-in connectors still resolve through the connector registry, not YAML specs
OAuth2 auto-refresh on 401 — oauth2 auth consumes a pre-fetched secrets.accessToken; the runtime does not refresh it
Filesystem auto-discovery of plugins — plugins must be imported and registered explicitly via connectorRegistry.registerPlugin(plugin)
Streaming connector plugins (Kafka, etc.) — the plugin SDK’s async-iterator read supports them, but none ship built-in

MCP Flow Server — createYamlConnector is now a real tool
Pipeline IR — source nodes reference connector ids registered here

Get Started

Tutorials

Use Cases

Decisioning

Studio

Data Pipelines

AI & ML

Operations & Reporting

Governance & Security

Integrations

Reference

YAML Connectors & Plugin SDK

Three tiers

YAML spec format

Auth types

Rate limit

HTTP runtime

Plugin SDK

MCP `createYamlConnector`

REST endpoints

AI generator

Not yet supported

​Three tiers

​YAML spec format

​Auth types

​Pagination types

​Rate limit

​HTTP runtime

​Plugin SDK

​MCP createYamlConnector

​REST endpoints

​AI generator

​Not yet supported

​Related

Three tiers

YAML spec format

Auth types

Pagination types

Rate limit

HTTP runtime

Plugin SDK

MCP `createYamlConnector`

REST endpoints

AI generator

Not yet supported

Related