Understand evaluations

evaluate and backtest are free and create no persistent model. Their REST routes are POST /v1/datasets/evaluate and POST /v1/forecast/backtest. The MCP tools are evaluate_data and backtest_forecast. Configure your client with Get started; these examples use churn.csv and an observed date,sales series in sales.csv.

Validation and assessmentCopy Anchor Link

validation describes whether the input is structurally usable. assessment is the API's comparison of the engine with its required references. Read assessment.status, assessment.metric and assessment.message together.

Display a better result as Better than the reference on this sample, alongside the API message and evaluated observation count. An improvement can happen by chance. No status certifies reliability, statistical significance, future performance or absence of predictive signal. High scores alone do not prove leakage; review field availability and independence separately.

Every successful evaluation response must include a non-null assessment. If it is absent, report a contract anomaly. Do not invent a status or substitute legacy signal or verdict fields. Preserve the API's conclusion unchanged.

Tabular evaluationCopy Anchor Link

report = spavik.evaluate("churn.csv", target="churn")
print(report["assessment"])
trial = report.get("trial")
if trial is not None:
    print(trial["metrics"], trial["baselines"])
    print(trial.get("n_test"), trial.get("evaluation"))
    print(trial.get("classification_report"))

Classification compares AUC; accuracy is secondary and cannot replace missing AUC. Regression compares MAE with a reference fitted on the training partition: trial.baselines.MAE.fit_on must be train. Each reference's score is in floor, for example trial.baselines.AUC.floor.

The engine uses a random holdout. trial.evaluation, when available, describes the split, original train/test row positions, sampling, class counts and AUC definition. Row positions are zero-based before dropping missing targets and exclude the header. trial.n_test is the test observation count; n_sampled_from counts eligible rows before sampling, while n_rows counts input rows.

Metrics by taskCopy Anchor Link

These scores describe held-out observations, not the unknown error of each new prediction. Secondary metrics do not change the primary assessment. There are no universal score thresholds for a model to be "good" or "usable".

For classification, evaluation.metric_context names the positive class and class order. AveragePrecision uses the non-interpolated precision-recall calculation, not trapezoidal PR-AUC. It is binary-only and is null when the positive class has no actual occurrences. LogLoss uses natural logarithms and clips probabilities at float64 machine epsilon. MacroF1 averages defined per-class F1 values equally; a class absent from both actual and predicted labels is excluded from this average, but remains visible in the per-class report.

A score that cannot be computed is null, with its reason in evaluation.metric_context.undefined_reasons. This includes missing or invalid probabilities, undefined AUC and R² on constant or insufficient actual values. Undefined is different from zero. Older engines may omit these additions.

The baseline for probability scores uses constant class frequencies fitted on the training partition. Accuracy and Macro F1 use the training majority class. Regression uses the training median for MAE and training mean for RMSE and R². All references are scored on the same held-out rows as the model. Training a saved model stores metrics, baselines, classification_report and evaluation on its version.

Compare the same metric on the same observations. MAE and RMSE retain the target's units. For example, a test MAE of 7 against a reference MAE of 10 can be presented as:

Mean absolute error: 7 units, compared with 10 for the reference. Observed error reduction: 30% on the test set.

This is an illustrative result, not a promised level of accuracy. baselines[metric].relative_error_reduction, when supplied, is the API's observed reduction: 0.3 means 30%, and a negative value means increased error. Read reduction_undefined_reason when it is null. If the field is absent, show the two scores without inventing a percentage. The SDKs and MCP do not calculate reductions or reconstruct the assessment.

A zero reference error makes the relative reduction undefined. Do not compare AUC to majority-class accuracy, apply error-reduction percentages to AUC or R², or turn 1 - MASE into a claimed improvement.

Classification errors by classCopy Anchor Link

trial.classification_report complements the numeric trial.metrics object. It does not change the AUC-based assessment. The same initial evaluation is saved as classification_report on model versions; older engines and versions may return null.

Each entry in per_class includes class, actual test support, predicted_count, true_positive, precision, recall, f1 and undefined_reasons. Support is the number of actual occurrences of that class in the evaluated test rows, not the number of predictions.

A class present but never predicted has precision null (no_predicted_samples), recall 0 and F1 0. A class absent from the test but predicted has precision 0, recall null (no_actual_samples) and F1 0. If both counts are zero, all three scores are null; F1 reports no_actual_or_predicted_samples. A defined score, including zero, has a null reason.

Read the returned decision_rule and class_order with the scores. They explain how the engine chose the predicted class, including any tie-breaking rule. Scores use the labels actually predicted on the same test rows as Accuracy. Do not recompute them from probabilities with another threshold. evaluated_count reports the number of rows evaluated.

Forecast backtestingCopy Anchor Link

report = spavik.backtest("sales.csv", target="sales", timestamp="date", horizon=7)
print(report["assessment"])
backtest = report.get("backtest")
if backtest is not None:
    print(backtest["metrics"], backtest["baselines"])
    print(backtest.get("n_eval"), backtest.get("evaluation"))
    print(backtest.get("intervals"))

These examples use the default mean point estimate and default quantiles. For custom settings, pass pointEstimate and quantiles in JavaScript, or point_estimate and quantiles in Python and REST.

Backtest the same prepared history, column roles, horizon, point_estimate and quantiles that you intend to use for the forecast. The API accepts mean (the default) or median. The result applies to the evaluated horizon and point estimate; it returns no forecast values.

The API compares MASE with both the naive reference and the historical-median constant reference on the same evaluation points. Use their returned labels and seasonal periods; do not compare MASE with a fixed threshold of 1.

With backtest.evaluation.protocol_version = 2:

• MASE uses a scale determined from each series' initial context.
• Eligible series receive equal weight, and evaluated points receive equal weight within each series.
• evaluation.mase_n_eval counts points included in MASE; backtest.n_eval counts all backtested points.
• Series with zero or unavailable scales are excluded from all MASE comparisons. Report the exclusions and reasons returned in evaluation.series.

Keep the API aggregates. Do not average the window scores yourself or infer v2 weighting when protocol metadata is missing. Window arrays show observed variation, not confidence intervals; a window index may represent different dates in different series.

Read prediction intervalsCopy Anchor Link

backtest.intervals reports nominal and observed coverage, the lower and upper quantile levels, average_width, width_unit: "target" and evaluated_count. A forecast requested with backtesting exposes the same object at intervals. It uses the widest available quantile pair and the same backtest observations for coverage and width.

Fractions remain between 0 and 1: 0.82 is displayed as 82%. Width is the mean upper bound minus lower bound, in the target's units. No percentage denominator is assumed. Read coverage and width together with the observation count; 82% coverage for an 80% target alone does not establish calibration. Greater coverage alone is not better: excessively wide intervals can cover almost everything. The current contract does not guarantee a Pinball baseline or an interval-quality verdict; report those measurements without inventing a reference comparison. Invalid or crossed bounds give null observed coverage and width with an undefined_reason; fewer than two quantiles give intervals: null.

Choose the next stepCopy Anchor Link

Inspect validation issues, the comparison, sample size and protocol limitations. Train or forecast when the available evidence fits your decision and its cost. Compare approaches only on matching periods, horizons, metrics and information available at each historical prediction cutoff.