Model feedback loop

A model can lose accuracy as data changes. Report outcomes to measure what happens and decide whether updating the model helps.

Follow Get started to configure spavik and create model. For REST, keep MODEL_ID from the training response. The examples below follow one customer prediction. Business identifiers stay outside the model features.

1. Keep the input and prediction in your applicationCopy Anchor Link

Every prediction carries a request_id. Save it together with the input row and result in your application. There is no prediction-history endpoint on any plan. Every outcome requires the original features and actual, regardless of data policy. request_id is optional correlation metadata. Keep the exact model version and serving engine too if you want to measure performance.

rows = [{"plan": "pro", "seats": 12, "tickets_90d": 3}]
result = model.predict(rows)
prediction_record = {
    "customer_id": "customer-123",
    "request_id": result.predictions[0]["request_id"],
    "features": rows[0],
    "result": result.predictions[0],
    "version": result.version,
    "engine": result.engine,
}
print(prediction_record)  # Save this mapping in your application.

Keep this record next to your own business identifier. The technical request log does not contain inference inputs or prediction values.

2. Report what happenedCopy Anchor Link

Report outcomes only after observing them. This example assumes the first customer's actual target value is 1, and a second observed example has value 0. Use your own observations and preserve the target's training vocabulary. Include the original features for every observed example so the batch works with every data policy. Add the saved prediction identifier when available.

outcomes = [
    {"request_id": prediction_record["request_id"],
     "features": prediction_record["features"], "actual": 1,
     "prediction": prediction_record["result"]["prediction"],
     "version": prediction_record["version"], "engine": prediction_record["engine"],
     "probabilities": {key[6:]: value for key, value in prediction_record["result"].items()
                       if key.startswith("proba_")}},
    {"features": {"plan": "free", "seats": 2, "tickets_90d": 10}, "actual": 0},
]
response = model.submit_outcomes(outcomes)
print(response.get("accepted"), response.get("pending_outcomes"))
print(response.get("refresh_triggered"), response.get("refresh_suggested"))

The response reports accepted and pending_outcomes. refresh_triggered is true when this batch already started an automatic integration, on a model with auto_update. refresh_suggested is true when enough outcomes are pending for a refresh to be worth considering. Invalid batches are rejected before ingestion.

3. Measure, before integratingCopy Anchor Link

Performance works the same for every data policy. Send the original prediction, exact version and engine with each outcome to populate observed and aggregate quality_reports; classification outcomes may also include probabilities. Outcomes without a supplied prediction can still refresh the model, but do not contribute to performance measurements.

The source is client_reported_predictions: these reports compare the predictions supplied by your application with its observed outcomes. The API never reads inference archives to calculate performance. Start with observed.assessment, then read the scores, matching baselines and evaluated counts. The same metrics and interpretation rules apply.

report = model.performance(days=30)
observed = report.get("observed")
if observed is None:
    print("No outcomes with original predictions to compare yet")
else:
    print(observed["assessment"])
    print(observed["metrics"], observed["baselines"])
    print(observed["metric_context"], observed["by_version"])
print(report.get("evaluation"), report.get("training_metrics"))
print(report.get("training_evaluation"), report.get("pending_outcomes"))

observed: null means no readable outcomes include their original predictions in this window. This is different from observed.assessment.status: "inconclusive", which reports insufficient comparison evidence in an observed evaluation. A populated observed without a valid assessment is a contract anomaly; do not replace it with a conclusion inferred from accuracy or an old verdict.

observed is null until outcomes include their original predictions. When available, it includes metric, compared, value, baseline, better_than_baseline and by_version. These existing summary fields use accuracy for classification and mean absolute error for regression. The full observed.metrics object follows the task's metric set, and observed.assessment compares AUC or MAE against the reference on these outcomes. Each by_version entry includes its own metrics, context, baselines and assessment.

Probability metrics use the probabilities recorded with the original predictions. They require valid probabilities for every compared row and compatible class orders across versions. Missing probabilities are not inferred from predicted labels. Scores are null with reasons in observed.metric_context.undefined_reasons; label-based metrics remain available. When class orders differ across versions, read each version's probability scores.

Baseline parameters come from each prediction version's training data and are scored again on these same outcomes. The reference is never fitted to the outcomes, and its original holdout score is not reused. Older versions without saved training parameters have null baselines with a reason. For MAE, RMSE and LogLoss, baselines[metric].relative_error_reduction is a fraction computed as 1 - model_score / floor, where floor is baselines[metric].floor, the reference's score; a zero baseline or unavailable score yields null.

The top-level evaluation reports the requested window start, comparison limit, counts fetched, readable and evaluated, and whether the limit was reached. training_evaluation retains the initial holdout context. All comparisons are descriptive; they do not establish future reliability.

Inspect the comparison count and coverage before drawing conclusions. A decline can have several causes; it does not by itself prove that a feature leaked the target. Review changes in data, feature availability and the prediction context. Python returns None and JavaScript returns null when performance analytics are unavailable; REST exposes the API error response.

For classification, observed.classification_report gives precision, recall and F1 by class with actual support, prediction counts and explicit reasons for undefined scores. Each entry of observed.by_version also carries its own classification_report, using that version's known classes. New labels found in outcomes are included.

These counts cover only readable predictions supplied with outcomes in the requested time window and configured comparison limit. They do not describe all predictions emitted. decision_rule.kind is recorded_prediction: the report evaluates the classes originally announced, without applying a new threshold. No supplied predictions means observed: null; regression reports have classification_report: null.

training_classification_report preserves the initial holdout evaluation for comparison. Refreshing a model retains that initial report and training metrics; it does not perform a fresh holdout evaluation.

4. Integrate pending outcomesCopy Anchor Link

refresh integrates pending outcomes into your model and produces a new version, which serves immediately.

response = model.refresh()
if response is None:
    print("No pending outcomes to integrate")
else:
    # version is the new ModelVersion object; its number is version["version"].
    print(response.get("version")["version"], response.get("outcomes_applied"))
    print(response.get("incomplete_outcomes"))

refresh returns None in Python or null in JavaScript only for NO_PENDING_OUTCOMES. Other API failures, including a refresh already in progress or an unavailable dependency, are raised. Inspect the error code before deciding the next action.

The previous version stays in the version history. Integration updates the examples and context retained by the model. incomplete_outcomes counts integrated outcomes missing some expected feature columns; those columns were included with empty values. If refresh_triggered was true, automatic integration has already started; do not request another refresh for that batch.

A reasonable rhythmCopy Anchor Link

There is no universal answer, but this pattern works well:

• report outcomes continuously, in batches, as your jobs run;
• measure once a week, and inspect the observed metric, count and baseline;
• consider integrating once a day, or when refresh_suggested turns true, after reviewing outcome quality and the observed assessment. A suggested refresh is not a promise of improved performance.

To integrate outcome batches automatically, enable automatic updates when creating your model. This creates a new model; it does not modify the existing one from the examples above.

model = spavik.train("churn.csv", target="churn", auto_update=True)

Getting notifiedCopy Anchor Link

Two webhook events exist for exactly this: model.stale when the model would benefit from an integration, model.refreshed when a new version is in service. See Webhooks.