Skip to main content

​
Datasets

Each dataset ships customers.csv, offers.json, outcomes.csv, splits.json, and meta.json. All synthetic — no PII. Generated by tools/qa/decisioning-bench/datasets/generate.ts with a hard-coded seed so every checkout reproduces the same rows.

​
Submission contract

A submission is a Docker image that:
  1. Listens on :8080.
  2. Accepts POST /recommend with body { customerId, channelId?, attributes? } and returns { decisionTraceId, offers: [{ offerId, score, rank }] }.
  3. Accepts POST /respond with body { customerId, outcome, ... }.
  4. Tolerates 100 RPS for 5 minutes. (The in-repo harness/run.mjs replays the holdout split as a correctness + latency smoke check; the sustained 100-RPS load is driven separately by platform/perf/recommend-100rps.ts.)

​
Running the harness

run.mjs writes one metrics file per run at results/<dataset>/<utc-timestamp>.json — latency percentiles (p50/p95/p99), rank-based AUC, per-group disparate-impact ratio, uplift-over-random, and error count. Leaderboard-CSV aggregation is not yet emitted by the harness (see Honest limits).

​
Scoring rubric

This is the intended rubric. The current in-repo run.mjs emits the raw per-dimension metrics (latency, AUC, fairness, uplift); it does not yet compute the weighted composite (normalized to 0-100) or score explanation quality — that aggregation lands with the leaderboard generator (see Honest limits).

​
Honest limits

  • V1 datasets are synthetic. Real-world distribution shift is not modeled — this is a capability check, not a market-fit signal.
  • Latency is measured from the harness on a single host. Multi-pod scale isn’t tested here; that’s k6’s job.
  • The tooling currently lives in tools/qa/decisioning-bench/. Splitting it out into a standalone internal service with a hosted leaderboard dashboard is operator-driven (needs a separate deployment).