🌐
IBM
developer.ibm.com › articles › implementing-logistic-regression-from-scratch-in-python
Implementing logistic regression from scratch in Python
Implement binary logistic regression from scratch in Python using NumPy. Learn sigmoid functions, binary cross-entropy loss, and gradient descent with real code.
Discussions

Logistic Regression in Python from Scratch
It’s good but you didn’t really discuss the mathematics like you said in the post. I guess it’d be better if you explained why we use the sigmoid and not another function with values between 0 and 1. Also, you could explain the choice of loss function. More on reddit.com
🌐 r/learnmachinelearning
6
83
September 5, 2020
Machine Learning Algorithms from Scratch in NumpY
If you want to implement in bare bone numpy. Check out this https://github.com/eriklindernoren/ML-From-Scratch More on reddit.com
🌐 r/learnmachinelearning
18
32
May 11, 2024
🌐
nick becker
beckernick.github.io › logistic-regression-from-scratch
Logistic Regression from Scratch in Python - nick becker
November 5, 2016 - I’ll add in the option to calculate the model with an intercept, since it’s a good option to have. def logistic_regression(features, target, num_steps, learning_rate, add_intercept = False): if add_intercept: intercept = np.ones((featur...
🌐
Medium
medium.com › @koushikkushal95 › logistic-regression-from-scratch-dfb8527a4226
Logistic Regression From Scratch. Logistic regression is often mentioned… | by Koushik Ahmed Kushal | Medium
August 28, 2023 - The model is simple and one of the easy starters to learn about generating probabilities, classifying samples, and understanding gradient descent. This tutorial walks you through some mathematical equations and pairs them with practical examples in Python so that you can see exactly how to train your own custom binary logistic regression model. Implementing logistic regression from scratch in Python
🌐
Analytics Vidhya
analyticsvidhya.com › home › implementing logistic regression from scratch using python
Implementing Logistic Regression from Scratch using Python
October 14, 2024 - This article went through different parts of logistic regression and saw how we could implement it through raw python code. But if you are working on some real project, it’s better to opt for Scikitlearn rather than writing it from scratch as it is quite robust to minor inconsistencies and less time-consuming.
🌐
Towards Data Science
towardsdatascience.com › home › latest › logistic regression from scratch in python
Logistic Regression From Scratch in Python | Towards Data Science
January 20, 2025 - First, we will understand the Sigmoid function, Hypothesis function, Decision Boundary, the Log Loss function and code them alongside. After that, we will apply the Gradient Descent Algorithm to find the parameters, weights and bias .
🌐
Dataquest
dataquest.io › blog › logistic-regression-in-python
An Intro to Logistic Regression in Python (100+ Code Examples)
January 7, 2025 - We started by building our own logistic regression model from scratch. We used our custom regression model to learn about convexity and the importance of choosing the approprate learning rate.
🌐
AskPython
askpython.com › home › logistic regression from scratch in python [algorithm explained]
Logistic Regression From Scratch [Algorithm Explained] - AskPython
August 6, 2022 - Logistic regression is a classic method mainly used for Binary Classification problems. even though it can be used for multi-class classification problems with some modification, in this article we will perform binary classification. Step by step we will break down the algorithm to understand its inner working and finally will create our own class. The sigmoid function in logistic regression returns a probability value that can then be mapped to two or more discrete classes.
Find elsewhere
🌐
GitHub
github.com › perborgen › LogisticRegression
GitHub - perborgen/LogisticRegression: Logistic regression from scratch in Python
Logistic regression from scratch in Python · This example uses gradient descent to fit the model. It also contains a Scikit Learn's way of doing logistic regression, so we can compare the two implementations.
Starred by 332 users
Forked by 273 users
Languages   Python 100.0% | Python 100.0%
🌐
Python Engineer
python-engineer.com › courses › mlfromscratch › 03_logisticregression
Logistic Regression in Python - ML From Scratch 03 - Python Engineer
In this Machine Learning from Scratch Tutorial, we are going to implement the Logistic Regression algorithm, using only built-in Python modules and numpy. We will also learn about the concept and the math behind this popular ML algorithm. All algorithms from this course can be found on GitHub together with example tests.
🌐
MachineLearningMastery
machinelearningmastery.com › home › blog › how to implement logistic regression from scratch in python
How To Implement Logistic Regression From Scratch in Python - MachineLearningMastery.com
December 11, 2019 - In this tutorial, you discovered how to implement logistic regression using stochastic gradient descent from scratch with Python.
🌐
Kaggle
kaggle.com › code › jagannathrk › logistic-regression-from-scratch-python
Logistic Regression from scratch - Python
Checking your browser before accessing www.kaggle.com · Click here if you are not automatically redirected after 5 seconds
🌐
Educative
educative.io › answers › implement-logistic-regression-in-python-from-scratch
Implement logistic regression in Python from scratch
Here’s an example of how to implement ... sklearn.model_selection import train_test_split · # Step 2: Load the dataset · data = load_breast_cancer() X, y = data.data, data.target ·...
🌐
Medium
dhirajkumarblog.medium.com › logistic-regression-in-python-from-scratch-5b901d72d68e
Logistic Regression Machine Learning Algorithm in Python from Scratch | by Dhiraj K | Medium
December 1, 2020 - Logistic Regression Machine Learning Algorithm in Python from Scratch Let’s understand the basics of Logistic Regression Introduction: When we are implementing Logistic Regression Machine Learning …
🌐
UVA Library
library.virginia.edu › data › articles › logistic-regression-four-ways-with-python
Logistic Regression Four Ways with Python | UVA Library
Fit the logistic regression model to the training dataset · Use the testing dataset with the model to predict testing dataset outcomes · Determine the accuracy of the model from these predictions · Again, it’s not always necessary to split your data into training and test sets, but it ...
Top answer
1 of 1
15

Solution

Just change the model creation line to

model = LogisticRegression(C=100000, fit_intercept=False)

Analysis of the problem

By default, sklearn solves regularized LogisticRegression, with fitting strength C=1 (small C-big regularization, big C-small regularization).

This class implements regularized logistic regression using the liblinear library, newton-cg and lbfgs solvers. It can handle both dense and sparse input. Use C-ordered arrays or CSR matrices containing 64-bit floats for optimal performance; any other input format will be converted (and copied).

Thus to obtain their model you should fit

model = LogisticRegression(C=1000000)

which gives

Intercept -2.038853 # this is actually half the intercept
study_hrs  1.504643 # this is correct

Furthermore the problem also lies in the way you work with data in patsy, see the simplified, correct example

import numpy as np
from sklearn.linear_model import LogisticRegression

X = [0.5,0.75,1.0,1.25,1.5,1.75,1.75,2.0,2.25,2.5,2.75,3.0,3.25,
3.5,4.0,4.25,4.5,4.75,5.0,5.5]
y = [0,0,0,0,0,0,1,0,1,0,1,0,1,0,1,1,1,1,1,1]

X = np.array([[x] for x in X])
y = np.ravel(y)

model = LogisticRegression(C=1000000.)
model = model.fit(X,y)

print('coef', model.coef_)
print('intercept', model.intercept_)

gives

coef [[ 1.50464059]]
intercept [-4.07769916]

What is the problem exactly? When you do dmatrices it by default embeds your input data with a column of ones (biases)

X = [0.5,0.75,1.0,1.25,1.5,1.75,1.75,2.0,2.25,2.5,2.75,3.0,3.25,
3.5,4.0,4.25,4.5,4.75,5.0,5.5]
y = [0,0,0,0,0,0,1,0,1,0,1,0,1,0,1,1,1,1,1,1]

zipped = list(zip(X,y))
df = pd.DataFrame(zipped,columns = ['study_hrs','p_or_f'])

y, X = dmatrices('p_or_f ~ study_hrs',
                  df, return_type="dataframe")

print(X)

which leads to

    Intercept  study_hrs
0           1       0.50
1           1       0.75
2           1       1.00
3           1       1.25
4           1       1.50
5           1       1.75
6           1       1.75
7           1       2.00
8           1       2.25
9           1       2.50
10          1       2.75
11          1       3.00
12          1       3.25
13          1       3.50
14          1       4.00
15          1       4.25
16          1       4.50
17          1       4.75
18          1       5.00
19          1       5.50

and this is why the resulting bias is just a half of the true one - scikit learns also added a column of ones... so you now have two biases, thus optimal solution is to give each of them half of the weight which would be given to a single one.

So what you can do?

  • do not use patsy in such a way
  • forbid patsy to add a bias
  • tell sklearn not to add bias

.

import numpy as np
import pandas as pd
from patsy import dmatrices
from sklearn.linear_model import LogisticRegression

X = [0.5,0.75,1.0,1.25,1.5,1.75,1.75,2.0,2.25,2.5,2.75,3.0,3.25,
3.5,4.0,4.25,4.5,4.75,5.0,5.5]
y = [0,0,0,0,0,0,1,0,1,0,1,0,1,0,1,1,1,1,1,1]

zipped = list(zip(X,y))
df = pd.DataFrame(zipped,columns = ['study_hrs','p_or_f'])

y, X = dmatrices('p_or_f ~ study_hrs',
                  df, return_type="dataframe")

y = np.ravel(y)

model = LogisticRegression(C=100000, fit_intercept=False)
model = model.fit(X,y)
print(pd.DataFrame(np.transpose(model.coef_),X.columns))

gives

Intercept -4.077571
study_hrs  1.504597

as desired

🌐
Visual Studio Magazine
visualstudiomagazine.com › articles › 2023 › 01 › 18 › logistic-regression.aspx
Logistic Regression from Scratch Using Raw Python -- Visual Studio Magazine
January 18, 2023 - Each weight cell is initialized to a random value between -0.01 and +0.01. The model bias is initialized to zero. An alternative design is to place the weights and bias into a Python class.
🌐
Real Python
realpython.com › logistic-regression-python
Logistic Regression in Python – Real Python
June 26, 2023 - In this step-by-step tutorial, you'll get started with logistic regression in Python. Classification is one of the most important areas of machine learning, and logistic regression is one of its basic methods. You'll learn how to create, evaluate, and apply a model to make predictions.