Skip to main content

​
What this solves

Regulated-finance, public-sector, and healthcare pilots need to prove that the decisioning system does not discriminate against protected groups. The hard-gate evaluates four fairness metrics (disparate impact ratio, demographic parity gap, equal opportunity gap, and Gini coefficient of decision-rate inequality), but reporting alone doesn’t stop a flow from going live when those metrics breach. The hard-gate is the enforcement layer.

​
How it works

Two enforcement points:
  1. Pre-publish gate — POST /api/v1/decision-flows/publish evaluates the policy against the last 7 days of decision traces. If any threshold is breached, the call returns 422 Unprocessable Entity with the violation list, and the flow stays in its previous status. The publish is blocked, not deferred.
  2. Continuous re-check cron — GET /api/v1/cron/fairness-recheck runs daily; for every tenant that has opted into continuousRecheck, it re-evaluates the gate and flips active flows back to paused when thresholds drift. Audit-logged with the violation reasons.
Override is via the existing four-eyes governance — set fairnessPolicy.override.{approvedBy, expiresAt} and the gate skips with an audit-logged warning until the override expires.

​
Step 1 — Configure the policy

  • disparateImpactRatio: 0.8 is the EEOC four-fifths rule (29 CFR § 1607.4D) — the lowest selection rate across protected groups must be at least 80% of the highest.
  • demographicParityGap: 0.2 blocks publishes where any pair of groups differs in selection rate by more than 20 percentage points.
  • equalOpportunityGap requires labeled outcomes (the customer would have converted); skipped automatically when labels aren’t available.
  • giniCoefficient: 0.2 bounds the inequality of decision rates across groups (OECD bands: <0.10 excellent, 0.10–0.20 acceptable, 0.20–0.40 monitor, >0.40 alert). The gate evaluates Gini with a default threshold of 0.2 even if you omit it, so set it explicitly to the band you want.
  • minSampleSize: 100 — the gate skips with a logged “insufficient samples” reason when fewer than 100 usable decision traces exist in the 7-day window.
  • continuousRecheck: true enables the post-publish drift cron.

​
Step 2 — Make sure the trace carries the sensitive attribute

The hard-gate reads decision_traces.requestAttributes[sensitiveAttribute] (added in manual-sql migration 22). Any recommend that passes the attribute in body.attributes is automatically eligible:
Without the attribute on the request, that trace becomes unusable for fairness sampling — the gate will skip with a “no usable samples” message until enough traces with the attribute exist.

​
Step 3 — Try to publish

When the gate fires:
The flow remains in its previous published state. Nothing rolls out.

​
Step 4 — Four-eyes override (when justified)

Documented exceptions are bypassed via the override block — both fields required:
While the override is active, every gate evaluation logs "Fairness gate bypassed via active override" with the approvedBy value — audit trail for the compliance team.

​
What gets auto-paused

When the continuous re-check cron detects drift on a tenant with continuousRecheck: true:
  • Every flow with status: "active" is updated to status: "paused".
  • A decision_flow audit-log row is written with action: "auto_pause", reason: "fairness_recheck_drift", and the violation list.
  • Operators see paused flows in Studio with the audit reason visible in the change history.
To re-enable, fix the upstream cause (model retrain, feature change, segment adjustment), let the next decision-trace window accumulate, and re-publish.

​
Gotchas

  • Sample sourcing requires manual-sql/22. Run psql ... -f prisma/manual-sql/22_decision_trace_request_attributes.sql once; this adds the requestAttributes JSONB column and a GIN index. Without it, the gate skips with "no usable samples".
  • The 7-day window is fixed. If your traffic is low, set minSampleSize accordingly or increase trace sampling rate (decisionTraceSampleRate).
  • continuousRecheck only auto-pauses currently-active flows. Re-publish brings them back; the gate evaluates again at that point.

​
Proof reference

T35 of the proof bundle (this section is forthcoming) will capture the full breach → block → override cycle with verbatim payloads.