Skip to main content

​
What ships

​
Running the 100 RPS scenario

The output is also printed to stdout so the operator can eyeball the percentiles before committing.

​
Publishing a baseline

  1. Run the scenario above against a stable environment (no concurrent dev work; warm caches). Numbers from a cold-start dev server are not representative of production.
  2. Inspect the JSON. Sanity-check latencyMs.p95 and totals.errorRate against your service’s SLOs.
  3. Commit the baseline file under platform/perf/baselines/.
  4. Open a PR. The CI step Perf-baseline regression gate (#24) will compare your new baseline against the previous one and fail the build if either threshold is breached.

​
CI gate semantics

platform/perf/compare-baselines.mjs reads the two most recent baseline files matching *-recommend-100rps.json (lexical sort → chronological because of the date prefix), parses them, and computes:
The build fails when either delta exceeds the configured threshold: Edge cases:
  • Zero baselines or directory missing. Pass with a [notice] log. First-time setup — the gate is a no-op until the operator publishes.
  • One baseline. Pass with a “no previous to compare” log.
  • Malformed JSON. Fail. A silent regression cannot slip through bad data.
  • Missing latencyMs.p95 or totals.errorRate field. Fail.
The gate is read-only — CI never generates a baseline (a CI runner is not load-representative; using it would yield noisy thresholds). Baselines are produced by operators against a representative environment.

​
Honest residual: 5K RPS is operator-pending

The user direction for #24 says explicitly: “5K RPS script ships but stays operator-pending (no AWS provisioning). If 5K RPS isn’t actually run, the baseline JSON file is operator-pending and the 3.1 grade reflects that honestly.” That is the case here. recommend-5krps.ts is the auditable load profile; running it reliably requires either:
  • A multi-node k6 cloud cluster (preferred — k6 already has the worker-pool primitives the TS script lacks for this scale), or
  • A self-hosted load fleet (one box per ~500 RPS budget).
When the 5K baseline is published, drop it under platform/perf/baselines/<date>-recommend-5krps.json and either run the gate manually with --scenario recommend-5krps or extend the CI step to invoke both scenarios.

​
Decisioning-bench --concurrency

tools/qa/decisioning-bench/harness/run.mjs now accepts an optional --concurrency N flag (default 1 = sequential, matching prior behaviour):
Internally this runs N async loops over a shared queue of holdout customers. Aggregation order remains deterministic — predictions is sorted by score before AUC + fairness computation. The output JSON gains a top-level concurrency: <N> field so downstream comparisons don’t conflate sequential vs parallel runs.

​
Roadmap

  • Auto-generate a [date]-recommend-100rps.json in CI on push to main against a long-running staging environment, so the gate catches regressions on every PR. Today’s manual-publish flow is the honest first cut.
  • Extend the gate to compare medians + p99 + actual achieved RPS, not only p95 + error rate. Trade-off is signal-vs-noise: short-window 100 RPS samples can show large p99 swings on cold caches.

​
Pipeline batch ingestion

Measured 2026-08-15 on a single local dev process against local Postgres — indicative shape, not a capacity guarantee. CSV → staging → append target, four columns, no transforms. Throughput rises with size as fixed per-run overhead amortises. All rows landed and were distinct at every size.
Loads above roughly 250,000 rows previously failed with 57014 canceling statement due to statement timeout and loaded zero rows, because the batch runtime inherited the decision hot path’s 8s statement cap. Batch loads now carry their own budget — see FLOW_BATCH_STATEMENT_TIMEOUT_MS.

​
Concurrent runs

Three simultaneous runs of the same pipeline over one file: exactly one run loads it, the others report that the file was claimed by a concurrent run. The atomic file move is the mutex, so a file is never processed twice and rows are never duplicated (verified: 500 rows in, 500 rows loaded, 500 distinct).