> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# decisioning-bench — NBA benchmark suite

> W10.2 — reproducible benchmark suite that pits any decisioning service against three synthetic datasets and a fixed scoring rubric.

## Datasets

| Dataset          | Customers | Offers | Domain                              |
| ---------------- | --------- | ------ | ----------------------------------- |
| `banking-cards`  | 5,000     | 12     | Credit-card propensity & cross-sell |
| `telco-churn`    | 8,000     | 6      | Retention offer routing             |
| `retail-loyalty` | 10,000    | 20     | Loyalty-tier upgrade & coupon       |

Each dataset ships `customers.csv`, `offers.json`, `outcomes.csv`,
`splits.json`, and `meta.json`. All synthetic — no PII. Generated by
`tools/qa/decisioning-bench/datasets/generate.ts` with a hard-coded
seed so every checkout reproduces the same rows.

## Submission contract

A submission is a Docker image that:

1. Listens on `:8080`.
2. Accepts `POST /recommend` with body `{ customerId, channelId?, attributes? }` and returns
   `{ decisionTraceId, offers: [{ offerId, score, rank }] }`.
3. Accepts `POST /respond` with body `{ customerId, outcome, ... }`.
4. Tolerates 100 RPS for 5 minutes. (The in-repo `harness/run.mjs` replays
   the holdout split as a correctness + latency smoke check; the sustained
   100-RPS load is driven separately by `platform/perf/recommend-100rps.ts`.)

## Running the harness

```bash theme={null}
cd tools/qa/decisioning-bench
docker run -d --name submission -p 8080:8080 my/submission:latest
node harness/run.mjs --dataset banking-cards --target http://localhost:8080
```

`run.mjs` writes one metrics file per run at
`results/<dataset>/<utc-timestamp>.json` — latency percentiles
(p50/p95/p99), rank-based AUC, per-group disparate-impact ratio,
uplift-over-random, and error count. Leaderboard-CSV aggregation is not yet
emitted by the harness (see **Honest limits**).

## Scoring rubric

| Dimension                     | Weight |
| ----------------------------- | ------ |
| Latency p99                   | 20%    |
| AUC (rank-based)              | 25%    |
| Fairness gap (worst DI ratio) | 15%    |
| Explanation quality           | 10%    |
| Uplift over random            | 30%    |

This is the *intended* rubric. The current in-repo `run.mjs` emits the raw
per-dimension metrics (latency, AUC, fairness, uplift); it does not yet
compute the weighted composite (normalized to 0-100) or score explanation
quality — that aggregation lands with the leaderboard generator (see
**Honest limits**).

## Honest limits

* V1 datasets are synthetic. Real-world distribution shift is **not**
  modeled — this is a capability check, not a market-fit signal.
* Latency is measured from the harness on a single host. Multi-pod
  scale isn't tested here; that's k6's job.
* The tooling currently lives in `tools/qa/decisioning-bench/`. Splitting
  it out into a standalone internal service with a hosted leaderboard
  dashboard is operator-driven (needs a separate deployment).