Learning cadence — how & how often models retrain

​

TL;DR

​

The three retraining mechanisms

​

1. Continuous (incremental)

• algorithmModel.modelState advances (e.g. Thompson arm’s α += 1 on positive, β += 1 on negative).
• algorithmModel.trainingSamples increments by 1.
• algorithmModel.lastLearnedAt is set to now.

​

2. Scheduled offline retrain

1. Finds every model with autoLearn: true, learnMode IN ("scheduled", "both"), and status IN ("active", "draft").
2. For each, retrains if EITHER the schedule elapsed (now - lastLearnedAt >= parseScheduleToMs(learnSchedule || "24h")) OR ~100+ new outcomes have accumulated since the last retrain (evidence co-trigger) — whichever fires first.
3. If due, runs a retrain pass inline (trainModelFromOutcomes) that recomputes weights / trees / embeddings from accumulated interaction-history rows.
4. On completion, writes new modelState, metrics, metricsHistory[], predictors[].importance, and bumps both lastTrainedAt and lastLearnedAt.

​

3. Drift-triggered retrain (primitives only — not auto-scheduled)

• computePSI / detectDrift (lib/model-governance.ts) compute a Population Stability Index and KL divergence on predictor score distributions. The default PSI trip threshold is 0.25 (PSI < 0.1 = stable; 0.1–0.25 = monitor; > 0.25 = drifted).
• runScheduledDriftChecks(tenantId) walks active models, compares recent interaction scores against each model’s metricsHistory baseline, and returns the set of drifted models to enqueue for retrain.

​

Toggling the cadence — API

# Turn on nightly retrains for a gradient-boosted model
curl -X PUT https://playground.kaireonai.com/api/v1/algorithm-models/$MODEL_ID \
  -H "Content-Type: application/json" \
  -d '{
    "autoLearn": true,
    "learnMode": "scheduled",
    "learnSchedule": "24h"
  }'

# Retrain every 6 hours instead
curl -X PUT https://playground.kaireonai.com/api/v1/algorithm-models/$MODEL_ID \
  -H "Content-Type: application/json" \
  -d '{ "learnSchedule": "6h" }'

# Pause auto-learning entirely (model continues to score, but state is frozen)
curl -X PUT https://playground.kaireonai.com/api/v1/algorithm-models/$MODEL_ID \
  -H "Content-Type: application/json" \
  -d '{ "autoLearn": false, "learnMode": "none" }'

​

Field reference

​

Toggling the cadence — UI

​

Recommended cadence per use case

• Cold-start with sparse historical data. Pick a thompson_bandit or epsilon_greedy. The continuous-learning behavior means the model is useful from request #1 — no warm-up needed. Move to bayesian after you have ~10k interactions and want to incorporate predictor features.
• Steady-state tabular workload with predictable churn. scorecard or gradient_boosted with autoLearn: true, learnMode: "scheduled", learnSchedule: "24h". Nightly is enough for most NBA workloads where the underlying customer behavior doesn’t shift hour-to-hour.
• Volatile distribution (pricing tests, seasonal pushes, regulatory changes). Same as above but drop to a "6h" (or shorter) cadence so the schedule absorbs sudden shifts. Automatic drift-triggered retrain is not wired today (§3), so a short schedule plus the ~100-outcome evidence co-trigger is how you stay fresh on volatile days. Watch PSI in Model Health to decide when to tighten the cadence further.
• Catalogs with fast turnover (new offers daily). Drop the schedule lower ("4h") or use drift-triggered exclusively — neural-cf especially benefits from fresh embeddings against the current catalog.
• Compliance-sensitive deployments where every retrain needs human review. Set autoLearn: false and use the model promotion endpoint (POST /algorithm-models/{id}/promote) to manually advance retrained candidates through draft → shadow → challenger → champion with audit-log review at each step.

​

Common questions