PYTHON / MACHINE LEARNING WITH PYTHON
Logistic regression and classification
Fit a logistic regression in scikit-learn, read its probabilities and log-odds, and choose a decision threshold on purpose.
What you will learn
- Fit LogisticRegression and read class probabilities with predict_proba
- Convert the linear score from decision_function into a probability with the sigmoid
- Locate the decision boundary as the point where w*x + b equals 0
- Pick a threshold other than 0.5 when false positives and false negatives differ in cost
Understanding Logistic regression and classification
Logistic regression computes exactly the same thing as a linear model, a weighted sum z = w1*x1 + ... + wn*xn + b, and then squashes z through the sigmoid 1/(1+exp(-z)) so the result always lands strictly between 0 and 1. That squashing is why it works for classification and a plain linear fit does not: a straight line predicting 0/1 labels happily outputs -0.4 or 1.7, which cannot be read as a probability, and its squared-error loss punishes a confidently correct point almost as much as a slightly wrong one. The sigmoid also means the model has no probability of exactly 0 or 1, only values that approach them as z runs off to minus or plus infinity.
Because z is linear, the boundary between the two classes sits where z = 0, that is where the probability is 0.5. With one feature that boundary is the single point x = -b/w; with two features it is a straight line, and with n features a flat hyperplane. This is the real limit of the method: it can only separate classes with a straight cut through the feature space you hand it, so a circular or XOR-shaped pattern is unlearnable until you add features such as squares or products that make it straight again.
The weights are found by maximising the likelihood of the observed labels, equivalently minimising log-loss, and unlike least squares this has no closed-form solution, so scikit-learn runs an iterative solver (lbfgs by default) until it converges. Two consequences matter in practice. First, LogisticRegression applies L2 regularization by default with C=1.0, so the coefficients you print are shrunk, not textbook maximum-likelihood values. Second, features on wildly different scales make the optimisation ill-conditioned and you will see a ConvergenceWarning; the fix is scaling the features, not blindly raising max_iter.
Finally, keep the model and the decision separate. predict_proba gives the probability, and predict is nothing more than that probability compared against 0.5. That 0.5 is a convention, not a property of the model, and on skewed or asymmetric-cost problems you should compare against your own threshold.
import numpy as np
from sklearn.linear_model import LogisticRegression
# one feature: hours studied. label 1 = passed the exam
hours = np.array([[0.5], [1.0], [1.5], [2.0], [2.5],
[3.0], [3.5], [4.0], [4.5], [5.0]])
passed = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
clf = LogisticRegression()
clf.fit(hours, passed)
w = clf.coef_[0, 0]
b = clf.intercept_[0]
test = np.array([[1.0], [2.7], [2.8], [4.5]])
manual = 1 / (1 + np.exp(-(w * test.ravel() + b)))
proba = clf.predict_proba(test)[:, 1]
print("boundary at hours =", round(-b / w, 2))
print("proba equals sigmoid(w*x+b):", bool(np.allclose(proba, manual)))
print("labels:", clf.predict(test))
print("labels equal proba >= 0.5:",
bool(np.array_equal(clf.predict(test), (proba >= 0.5).astype(int))))Logistic regression is a linear score passed through the sigmoid, so it produces probabilities and a straight decision boundary at z = 0.
Worked examples
The raw score is the log-odds
Shows that decision_function returns log(p/(1-p)) and that predict is just the sign of that score.
import numpy as np
from sklearn.linear_model import LogisticRegression
X = np.array([[2.0], [3.0], [4.0], [5.0], [6.0], [7.0]])
y = np.array([0, 0, 0, 1, 1, 1])
clf = LogisticRegression().fit(X, y)
z = clf.decision_function(X)
p = clf.predict_proba(X)[:, 1]
print("z is log(p/(1-p)):", bool(np.allclose(z, np.log(p / (1 - p)))))
print("predict is z > 0:", bool(np.array_equal((z > 0).astype(int), clf.predict(X))))
print("z increases with x:", bool(np.all(np.diff(z) > 0)))Example explained
Line 1decision_function returns w*x + b before any squashing, which is the log-odds of the positive class.
Line 2np.log(p / (1 - p)) inverts the sigmoid, so it reconstructs z from the probability exactly.
Line 3Comparing z to 0 is the same test as comparing p to 0.5, which is why predict needs no probabilities.
Line 4z rises monotonically with x because the fitted coefficient is positive, so probability is monotone in the feature too.
Three classes with softmax
Shows how the same estimator handles more than two classes by learning one weight vector per class.
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
# fitting on everything here only to inspect shapes
clf = LogisticRegression(max_iter=1000).fit(X, y)
p = clf.predict_proba(X[:3])
print("classes:", clf.classes_)
print("coef shape:", clf.coef_.shape)
print("row sums:", np.round(p.sum(axis=1), 6))
print("argmax equals predict:",
bool(np.array_equal(clf.classes_[p.argmax(axis=1)], clf.predict(X[:3]))))Example explained
Line 1coef_ is (3, 4): one row of four feature weights per class, so there are three linear scores per sample.
Line 2The three scores go through softmax rather than a single sigmoid, which forces every row of predict_proba to sum to 1.
Line 3predict picks the class with the largest probability, so it equals classes_ indexed by argmax.
Line 4max_iter=1000 is needed because the raw iris measurements are unscaled and lbfgs would otherwise stop early with a warning.
Important notes
The boundary is linear in whatever features you supply, so a curved separation requires you to add terms such as x**2 or x1*x2 yourself.
For coefficients close to unregularized maximum likelihood, pass penalty=None (or a very large C), but on perfectly separable data the weights then diverge and the fit will not settle.
Common mistakes
Reading predict_proba(X)[:, 0] as the chance of the positive class; column 0 is the first entry of clf.classes_, so every probability comes out inverted and the model looks worse than random.
Assuming the default fit is plain maximum likelihood and interpreting coefficients directly; LogisticRegression uses L2 with C=1.0, so weights are shrunk, and on unscaled features the shrinkage hits the small-scale feature hardest.
Judging an imbalanced problem only by predict at the implicit 0.5 threshold, which can label everything the majority class while the probabilities still rank the positives correctly.
Try it yourself
Change, predict, then run
Fit LogisticRegression on the hours/passed data from the main example, print predict_proba for hours 2.5 to 3.0 in steps of 0.05, and confirm that the first value at or above 0.5 sits at the boundary -intercept_/coef_.
Open the Python workspaceCheck your understanding
A fraud classifier trained with logistic regression flags too few fraudulent transactions, and you are willing to accept more false alarms. Which change achieves that most directly?
- Refit the model with a different random_state so the solver finds different weights
- Compare predict_proba against a threshold below 0.5 instead of calling predict
- Increase C to weaken regularization so the coefficients grow larger
- Swap the sigmoid for softmax so the probabilities sum to 1
Show answer
predict is only predict_proba compared to 0.5, so lowering that cutoff labels more transactions as fraud without retraining anything. Raising C changes how much the weights are shrunk and can shift probabilities in either direction; it is a fitting choice, not a control over the trade-off between missed fraud and false alarms.