A/B test with holdout

​

What this solves

​

Why this works

1. Champion / Challenger on the Score node — championChallenger.{champion, challengers[]} routes each customer to one scoring model via a deterministic hash of customerId, so the same customer always lands in the same variant. The assignment is persisted (see Step 3).
2. Per-flow holdout — a flow-level experiment config block ({ enabled: true, holdoutPercent: N }). Held-out customers get zero offers and the trace carries traceSummary.holdout: true. This is a true causal holdout — the customer sees nothing from this flow, so you can measure lift against “did nothing.”
3. Always-on control group — an independent ~2% slice (deterministic per customer × day) that still receives the qualified, policy-filtered offer set but with randomized scores/ranking instead of the model’s ranking. Surfaced as controlGroup: true on every response.

​

Step 1 — Reserve a holdout on the flow

{
  "experiment": {
    "enabled": true,
    "holdoutPercent": 10,
    "controlGroupAction": "no_offers"
  }
}

​

Step 2 — Configure champion/challenger on the Score node

{
  "id": "score",
  "type": "score",
  "config": {
    "method": "formula",
    "championChallenger": {
      "enabled": true,
      "experimentId": "cards-q4-uplift",
      "champion":    { "modelKey": "scorecard-v2",   "weight": 50 },
      "challengers": [
        { "modelKey": "bayesian-v2",         "weight": 30 },
        { "modelKey": "gradient_boosted-v2", "weight": 20 }
      ]
    }
  }
}

​

Step 3 — Capture the variant on each decision

{
  "experimentVariant": "bayesian-v2",
  "controlGroup": false,
  ...
}

​

Step 4 — Measure uplift

curl https://playground.kaireonai.com/api/v1/experiments/cards-q4-uplift/results \
  -H "X-Requested-With: XMLHttpRequest"

​

Gotchas

• Holdout is per-flow, not tenant-wide. It lives on the flow’s experiment.holdoutPercent, so different flows can hold out different shares. The always-on control group, by contrast, is tenant-wide: it reads tenant.settings.controlGroupPercent and defaults to 2% when unset. (That default is what applies unless the value is seeded directly on the tenant record — controlGroupPercent is not on the PUT /api/v1/tenant-settings allowlist.)
• Variant assignment is persistent. The same customer always sees the same variant — a DB-backed assignment (variant_assignments, 30-day TTL) keyed by experimentId, falling back to a deterministic hash on a miss.
• autoPromote on the Experiment resource (when enabled) automatically promotes a winning challenger to champion once it clears promoteThreshold for promoteAfterDays. Combine with four-eyes approval for governance.

​

What the trace will show

customerId   | experimentVariant | controlGroup | holdout | finalCount
cust-A-001   | scorecard-v2      | false        | false   | 3
cust-A-002   | bayesian-v2       | false        | false   | 3
cust-A-003   | scorecard-v2      | true         | false   | 3   ← control group: offer set kept, ranking randomized
cust-A-004   | (none)            | false        | true    | 0   ← per-flow holdout: zero offers, traceSummary.holdout=true

​

Proof reference