The feedback loop

A model trained once decays as reality drifts away from its examples. The feedback loop is what prevents that.

predict  ->  reality happens  ->  submit_outcomes  ->  performance  ->  refresh  ->  predict

1. Keep each prediction's identifier

Every prediction carries a request_id. It is what lets you later attach what actually happened without re-describing the row.

r = model.predict(rows)
for row, prediction in zip(rows, r.lignes):
    store(customer_id=row["id"], request_id=prediction["request_id"])

Keep that request_id next to your own business identifier. One column in your database is enough.

2. Report what happened

Two forms, and you can mix them in the same call.

model.submit_outcomes([
    # by identifier: the API looks the features up on its own
    {"request_id": "1bc55e5c-...", "actual": 1},

    # by features: when you did not keep the identifier
    {"features": {"plan": "free", "seats": 2, "tickets_90d": 10}, "actual": 0},
])

actual holds the value that was actually observed, in the same vocabulary as your training target. If your target was 0 or 1, actual is 0 or 1.

The response tells you what was accepted, what is pending, and what could not be matched:

r = model.submit_outcomes(outcomes)
r.get("accepted")            # 3
r.get("pending_outcomes")    # 3, awaiting integration
r.get("unresolved")          # those no prediction matches
r.get("refresh_suggested")   # True when integrating becomes worthwhile

Batch your outcomes

One call is one batch. On a model with auto_update, each batch triggers an integration: sending outcomes one at a time would trigger one integration per outcome.

3. Measure, before integrating

performance compares what was predicted to what happened, and says in one sentence what to make of it.

p = model.performance(days=30)
p.get("verdict")            # {'level': 'holding', 'message': 'The model does better than the lazy answer on real outcomes.'}
p.get("training_metrics")   # what training announced
p.get("observed")           # what is observed, broken down by version
p.get("pending_outcomes")   # outcomes not yet integrated

The verdict has three levels. holding: the model beats the lazy answer on real outcomes, integrate. weak: it does no better than always answering the majority class, add outcomes or train on columns known before the event. dropped: real accuracy is far below training, which almost always means a column filled in after the event looked like a good feature. In that case do not integrate: find the column, retrain without it. The verdict is absent for a regression, which has no measured floor.

observed stays null as long as no outcome can be matched to a prediction: normal in the first days. The response also breaks performance down by version, which answers the only question that really matters: did the last integration improve things, or not.

4. Integrate, if it holds

refresh folds pending outcomes into the context data and produces a new version, which serves immediately.

r = model.refresh()
if r is None:
    print("nothing to integrate right now")
else:
    print(r.get("version")["version"], r.get("outcomes_applied"))

refresh returns None when nothing is pending. That is a normal state, not an error: you can call it every night without checking whether there is work.

The previous version stays readable, so rolling back is possible. And the weights do not change: it is the set of examples that grows.

A reasonable rhythm

There is no universal answer, but this pattern works well:

• report outcomes continuously, in batches, as your jobs run;
• measure once a week, and read the verdict before anything else;
• integrate once a day, or as soon as refresh_suggested turns true, as long as the model holds.

If you would rather not think about it, create the model with auto_update=True: batches trigger the integration themselves.

Getting notified

Two webhook events exist for exactly this: model.stale when the model would benefit from an integration, model.refreshed when a new version is in service. See Webhooks.