PYTHON / MACHINE LEARNING WITH PYTHON
Saving models and reporting results honestly
Persist a fitted scikit-learn pipeline with joblib, reload it safely, and report a test score that is not inflated by tuning.
What you will learn
- Save the whole Pipeline with joblib, never the bare estimator
- Verify a reloaded model reproduces the original predictions exactly
- Quote a test score once, next to a DummyClassifier baseline
- Store feature order, metric and sklearn version beside the model file
Understanding Saving models and reporting results honestly
A fitted model is two things at once: learned parameters and the transformations those parameters expect. StandardScaler stores the training mean and standard deviation, LogisticRegression stores coefficients tuned to that scaled space. If you save only the classifier, the coefficients are meaningless against raw input, so joblib.dump should always be pointed at the whole Pipeline object, which pickles the scaler's fitted statistics along with the coefficients.
joblib is pickle with a faster path for large NumPy arrays, so it writes a Python object graph, not a portable model description. That has two consequences: loading a file runs pickle's reconstruction machinery, so you only load files you produced or trust, and the file is bound to the class definitions that created it. Upgrading scikit-learn can make a load emit InconsistentVersionWarning or, worse, silently reconstruct an object whose internals changed meaning, which is why the version belongs in the file next to the model.
Honest reporting follows the same logic as saving: a number is only as good as the data that produced it. Every time you look at the test set to choose a threshold, a hyperparameter, or a model family, you have fitted something to it, and the score stops being an estimate of future performance and becomes a best-of-N maximum. Make choices with cross-validation on the training split, touch the test set once at the end, and print it beside the majority-class baseline and the test set size so a reader can see whether 0.95 is impressive or trivial.
import os
import tempfile
import joblib
import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = np.concatenate([rng.normal(-4, 0.5, 100), rng.normal(4, 0.5, 100)]).reshape(-1, 1)
y = np.array([0] * 100 + [1] * 100)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
pipe = Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression(max_iter=1000))])
pipe.fit(X_tr, y_tr)
path = os.path.join(tempfile.gettempdir(), "model.joblib")
joblib.dump(pipe, path)
reloaded = joblib.load(path)
print("same predictions after reload:", np.array_equal(pipe.predict(X_te), reloaded.predict(X_te)))
print(f"test accuracy (used once): {reloaded.score(X_te, y_te):.3f}")
baseline = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
print(f"majority-class baseline: {baseline.score(X_te, y_te):.3f}")
print("test set size:", len(y_te))
os.remove(path)A saved model must carry its preprocessing with it, and a reported score is only unbiased if the data behind it influenced no decision.
Worked examples
Saving the estimator but losing the scaler
Shows what happens when the fitted preprocessing is not persisted with the model.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(1)
X = np.concatenate([rng.normal(100, 0.5, 50), rng.normal(108, 0.5, 50)]).reshape(-1, 1)
y = np.array([0] * 50 + [1] * 50)
scaler = StandardScaler().fit(X)
clf = LogisticRegression(max_iter=1000).fit(scaler.transform(X), y)
print(f"scaled input: {clf.score(scaler.transform(X), y):.3f}")
print(f"raw input: {clf.score(X, y):.3f}")
print("predicted classes on raw data:", set(clf.predict(X).tolist()))Example explained
Line 1The two classes sit near 100 and 108, so standardizing maps them to roughly -1 and +1 around zero.
Line 2clf learned a positive coefficient and an intercept near zero, which is correct only in that centred space.
Line 3Fed raw values of 100 and above, the decision function is positive for every row, so every prediction is class 1.
Line 4Accuracy collapses to 0.500 with no exception raised: the shapes match, only the units are wrong.
Bundling metadata with the model
Stores the model together with the facts a reader needs to interpret its reported score.
import os
import tempfile
import joblib
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(2)
X = np.concatenate([rng.normal(-3, 0.4, 60), rng.normal(3, 0.4, 60)]).reshape(-1, 1)
y = np.array([0] * 60 + [1] * 60)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=1)
model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
bundle = {
"model": model,
"feature_names": ["signal"],
"metric": "accuracy",
"score": float(model.score(X_te, y_te)),
"n_train": int(len(y_tr)),
"n_test": int(len(y_te)),
}
path = os.path.join(tempfile.gettempdir(), "bundle.joblib")
joblib.dump(bundle, path)
loaded = joblib.load(path)
print(sorted(loaded))
print(f"{loaded['metric']}: {loaded['score']:.3f} on {loaded['n_test']} held-out rows")
print("expects features:", loaded["feature_names"], "| model sees", loaded["model"].n_features_in_)
os.remove(path)Example explained
Line 1joblib.dump accepts any picklable object, so a plain dict lets the model travel with its own documentation.
Line 2score is computed once on X_te and frozen into the file, so nobody is tempted to recompute it after further tuning.
Line 3float() and int() strip NumPy scalar types, keeping the metadata readable from other tools.
Line 4n_features_in_ comes from the fitted estimator and can be checked against feature_names to catch a column mismatch at load time.
Important notes
joblib.load runs pickle, which can execute arbitrary code, so treat model files like executables and only load ones from a source you trust.
A .joblib file is not an archival format; keep the training script and the data recipe so the model can be rebuilt when the library moves on.
Common mistakes
Pickling only the classifier and rescaling by hand at prediction time: the hand-written scaling uses different means, and predictions are wrong without any error.
Reporting the best cross-validation score reached while tuning as the expected accuracy: that maximum is optimistic by construction, and live performance comes out lower.
Loading a model into a newer scikit-learn than it was saved with: attributes may be renamed or reinterpreted, so the load either warns or produces a subtly different model.
Quoting a bare accuracy on imbalanced data: 0.98 can be worse than predicting the majority class, and without the baseline nobody can tell.
Try it yourself
Change, predict, then run
Train a Pipeline of StandardScaler and LogisticRegression on a stratified train split, dump it with joblib, load it in the same script and assert the reloaded predictions are identical. Then print the test accuracy next to a DummyClassifier(strategy="most_frequent") score and the test set size.
Open the Python workspaceCheck your understanding
You compare 20 hyperparameter settings by scoring each on the test set, keep the best, and publish that score. What is wrong with the published number?
- It is optimistically biased, because the test set influenced the choice and is no longer independent of the model
- Nothing, since the test set was never passed to fit() and the model never saw those labels during training
- It is only a problem when the test set is smaller than the training set
- It is fine because all 20 settings were judged on identical data, so the comparison is fair
Show answer
Taking the maximum of 20 noisy estimates on the same data selects partly for luck on that data, so the winning score overstates future performance. Option 2 is tempting because fit() really never touched the test set, but selection is a form of fitting: choosing on the test set uses its labels just as surely as gradient descent would. Tune with cross-validation on the training split and keep the test set for one final measurement.