Skip to main content

​
What it does

A nightly script — tools/scripts/run-negotiation-eval.ts, run from the platform/ working directory — drives the negotiation eval harness (runDefaultNegotiationEval) with a fixed seed (default 7), prints a JSON summary, and writes the full report to platform/perf/baselines/<date>-negotiation-eval.json. The script’s exit code is the regression signal:

​
When it runs

.github/workflows/nightly-eval.yml schedules the script at 06:17 UTC daily. The cron offset (17 minutes past) deconflicts with other cron-driven jobs in the deployment. workflow_dispatch: is also enabled so operators can re-run on demand.

​
What gets archived

Each run uploads platform/perf/baselines/*-negotiation-eval.json as the workflow artifact negotiation-eval-<run_id> with 30-day retention. Operators can compare today’s report against a 7-day-prior artifact to spot drift even when the daily exit code stays green.

​
Honest limits

  • Synthetic-only dataset. Production negotiation transcripts are not fed into this harness — they would expose customer PII and require red-team review before use.
  • On failure the workflow posts to Slack via its Notify Slack on failure step when the SLACK_WEBHOOK_URL repo secret is set; when the secret is absent that step is skipped and operators fall back to the standard GitHub Actions failure email. The eval script itself never POSTs anywhere — notification lives entirely in the workflow.
  • Default seed is 7, set as the NEGOTIATION_EVAL_SEED env on the workflow’s run step. It is not exposed as a workflow_dispatch input — to investigate a flake, run the script locally with a different value (NEGOTIATION_EVAL_SEED=42 npx tsx ../tools/scripts/run-negotiation-eval.ts) or edit that env in the workflow.

​
Cross-references

  • Negotiation guardrail engine: lib/negotiation/guardrails.ts.
  • Apply-mode + multi-turn (lib/negotiation/apply-mode.ts, lib/negotiation/multi-turn.ts) are implemented and wired into the recommend hot-path (lib/negotiation/realtime-apply.ts) plus a dedicated POST /api/v1/decisions/:id/negotiate/apply route — but stay behind default-off gates: the tenant apply-mode flag, offer-level negotiable, guardrail validation, a two-sided kill switch, a per-tenant/offer apply-rate budget, and optional regulator-review sign-off.