> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# YAML Connectors & Plugin SDK

> Declarative HTTP connectors via YAML; engineer-authored plugins via the typed SDK; AI-generated drafts from an API docs URL.

KaireonAI supports a three-tier connector model for wiring HTTP/REST sources beyond
the built-in registry. All three tiers are wired end-to-end: server-side validation,
REST + MCP registration, and an authoring page at **Data → YAML Connectors** (paste a
spec, validate, register — or generate a draft with the AI generator).

<Warning>
  **Specs are validated, not yet runnable.** The registry is **in-memory**, so a
  registered spec does not survive a server restart or redeploy, and registering
  creates **no connector record**. A YAML connector therefore never appears in the
  New Pipeline connector picker and cannot be used as a pipeline source — the IR
  `source` node accepts a closed set of file/object-storage kinds, which a YAML
  spec is not among. Use this tier to author and validate specs; ingestion for
  YAML connectors is not wired yet.
</Warning>

## Three tiers

| Tier                  | Mechanism                                                             | When                                                        |
| --------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------- |
| **YAML spec**         | Declarative HTTP/REST connector                                       | 90% of SaaS APIs (Salesforce, Stripe, HubSpot, Klaviyo, …)  |
| **TypeScript plugin** | Typed plugin SDK with a single `defineConnector` factory              | Streaming + complex protocols (Kafka, Snowflake COPY, JDBC) |
| **AI-generated YAML** | AI drafts a spec from a docs URL, OpenAPI snippet, or plain-text hint | Net-new HTTP APIs you'd rather not hand-author              |

## YAML spec format

```yaml theme={null}
kind: connector
version: "1.0"
id: salesforce_rest
displayName: Salesforce REST
category: crm
auth:
  - type: oauth2
    authorizeUrl: https://login.salesforce.com/services/oauth2/authorize
    tokenUrl: https://login.salesforce.com/services/oauth2/token
    scopes: [api, refresh_token]
    clientIdRef: "{{secrets.clientId}}"
    clientSecretRef: "{{secrets.clientSecret}}"
endpoints:
  - id: query
    method: GET
    url: "{{instance_url}}/services/data/v59.0/query"
    params:
      q: "{{soql}}"
    pagination:
      type: cursor
      cursorField: nextRecordsUrl
      pageSize: 200
    rateLimit:
      requestsPerSecond: 10
read:
  primary: query
  resourceTypes: [Account, Contact, Lead, Opportunity]
```

Validated by the connector-YAML parser — js-yaml load + Zod safeParse, same two-phase pattern as the pipeline-IR parser. Invalid specs are rejected before they enter the registry.

### Auth types

`none`, `api_key` (header or query), `basic`, `oauth2` (operator passes a pre-fetched accessToken in `secrets.accessToken`; Phase 5 doesn't auto-refresh, that's a follow-up).

### Pagination types

`none`, `cursor` (follows a response field until absent), `offset` (numeric offset+limit), `page` (page number). Each stops on a short page or `maxPages` cap (default 100).

### Rate limit

`requestsPerSecond` paces inter-page sleeps. `burstSize` is reserved for a future token-bucket implementation.

## HTTP runtime

The YAML endpoint executor runs the call against the named spec, given the endpoint id, request parameters, secrets, and an abort signal:

1. Resolve auth → headers + query params
2. Substitute `{{var}}` templates against (params + secrets)
3. SSRF-validate the URL via `lib/security/url-validator.validateAndResolve`
4. Fetch with rate-limit pacing
5. Walk pagination
6. Extract rows via `responseRowsPath` (defaults to root)

## Plugin SDK

```ts theme={null}
import { z } from "zod";
import { defineConnector } from "@/lib/flow/connectors/plugin-sdk";

export default defineConnector({
  id: "kafka",
  displayName: "Apache Kafka",
  category: "streaming",
  configSchema: z.object({
    brokers: z.array(z.string()),
    topic: z.string(),
    consumerGroup: z.string(),
  }),
  read: async function* (ctx) {
    // implementation; yields { key, value, partition, offset, ... }
  },
  testConnection: async (config) => ({ ok: true }),
});
```

Plugins register themselves in the in-memory `connectorRegistry` via `connectorRegistry.registerPlugin(plugin)`. Plugin imports must be explicit (no filesystem auto-discovery in Phase 5).

## MCP `createYamlConnector`

Promoted from Phase 2b stub. Accepts a YAML text body, validates via `parseConnectorYaml`, registers on success.

```json theme={null}
{
  "ok": true,
  "id": "salesforce_rest",
  "displayName": "Salesforce REST"
}
```

Or:

```json theme={null}
{
  "ok": false,
  "errors": ["auth.0.tokenUrl: must be https://"]
}
```

## REST endpoints

Two REST routes back the same in-memory registry. Both require an `admin` or `editor`
role plus a tenant; the authoring page posts to them.

| Route                          | Purpose                                      | Response                                                                                                       |
| ------------------------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `POST /api/v1/connectors/yaml` | Validate + register a spec — body `{ yaml }` | `201 { ok, id, displayName, category }`; `400 { ok: false, errors }` on invalid YAML; `409` on an id collision |
| `GET /api/v1/connectors/yaml`  | List registered YAML + plugin connectors     | `200 [{ id, displayName, category, kind }]`                                                                    |

A registered spec's `id` is then available to Flow IR source nodes.

## AI generator

`POST /api/v1/ai/generate-yaml-connector` drafts a spec for you. Body:
`{ docsUrl?, openapi?, hint? }` — at least one is required. The generator fetches the
docs (SSRF-guarded), prompts the tenant's configured model via `generateObject` against
the connector-spec Zod schema, and retries up to twice on validation failure —
re-prompting each time with the structured errors.

```json theme={null}
{
  "ok": true,
  "retries": 0,
  "tokensUsed": 1830,
  "spec": { "kind": "connector", "version": "1.0", "id": "brevo", "...": "..." },
  "yaml": "kind: connector\nversion: \"1.0\"\n..."
}
```

On failure it returns `{ ok: false, errors, retries, tokensUsed }` (HTTP 200). The
route requires an `admin`/`editor` role and the tenant's **Flow IR** feature flag; the
generator panel is collapsible on the **Data → YAML Connectors** page.

## Not yet supported

* Marketplace UI — a browse/install catalog for shared specs (distinct from the authoring page, which ships today)
* Bulk migration of the built-in registry's HTTP-shaped connectors to YAML specs — the 80 built-in connectors still resolve through the connector registry, not YAML specs
* OAuth2 auto-refresh on 401 — `oauth2` auth consumes a pre-fetched `secrets.accessToken`; the runtime does not refresh it
* Filesystem auto-discovery of plugins — plugins must be imported and registered explicitly via `connectorRegistry.registerPlugin(plugin)`
* Streaming connector plugins (Kafka, etc.) — the plugin SDK's async-iterator `read` supports them, but none ship built-in

## Related

* [MCP Flow Server](/ai-ml/mcp-flow-server) — `createYamlConnector` is now a real tool
* [Pipeline IR](/data/transforms/pipeline-ir) — source nodes reference connector ids registered here