Linear Regression

Linear Regression is one of the simplest yet powerful algorithms in machine learning and statistics. It’s widely used in predictive modeling, where the goal is to understand the relationship between one or more independent variables and a dependent variable. The power of linear regression lies in its simplicity and interpretability, making it a go-to model for many data scientists.

In this post, we will explore the core concepts of linear regression, its mathematical foundation, how to implement it in Python, the role of regularization techniques like Ridge and Lasso, and finally, evaluate the model using metrics like Mean Squared Error and R-squared.

What is Linear Regression?
Mathematical Foundation of Linear Regression
Ordinary Least Squares (OLS) Method
Implementing Linear Regression in Python using Scikit-learn
Regularization Techniques: Ridge (L2) and Lasso (L1) Regression
Evaluation Metrics for Linear Regression: Mean Squared Error and R-squared
Case Study: Predicting House Prices

What is Linear Regression?

Linear regression is a statistical method used to model the relationship between a dependent variable (target) and one or more independent variables (predictors). The idea is to fit a straight line that best describes the data points.

In its simplest form, linear regression is a straight-line model defined by the following equation:

$y = \beta_0 + \beta_1 x + \epsilon$

Where:

$y$ is the dependent variable (what you’re predicting).
$x$ is the independent variable (the predictor).
$\beta_0$ is the intercept.
$\beta_1$ is the slope (how much $y$ changes when $x$ changes).
$\epsilon$ is the error term, representing the difference between actual and predicted values.

Linear regression can be extended to handle multiple predictors, leading to multiple linear regression, where:

$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_n x_n + \epsilon$

Mathematical Foundation of Linear Regression

The foundation of linear regression lies in fitting a straight line through the data points that minimizes the difference between the observed and predicted values. Mathematically, the predicted values are represented as:

$\hat{y} = \beta_0 + \beta_1$

The goal of the linear regression algorithm is to estimate the values of $beta_0$ and $\beta_1$ such that the total error (residual) is minimized. The error is usually measured using the sum of squared residuals (difference between actual $y$ and predicted $\hat{y}$ ):

$J(\beta_0, \beta_1) = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

Where $x$ is the number of data points. This method is known as Ordinary Least Squares (OLS).

Ordinary Least Squares (OLS) Method

The OLS method estimates the parameters of a linear regression model by minimizing the sum of the squared residuals. It aims to find the line (or hyperplane in multiple linear regression) that minimizes the following cost function:

$J(\beta) = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

Where:

$y_i$ is the actual value of the dependent variable.
$\hat{y}_i$ is the predicted value using the model.
$n$ is the number of data points.

The solution to this optimization problem gives us the best-fitting line for the data.

Implementing Linear Regression in Python using Scikit-learn

The Python library Scikit-learn provides an easy-to-use interface to implement linear regression. Here’s a step-by-step guide to building a linear regression model.

Step 1: Import the necessary libraries

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

Step 2: Load the dataset

For this example, let’s assume we have a dataset with house features and prices:

data = pd.read_csv('housing_data.csv')

Step 3: Preprocess the data

Separate the features (independent variables) and target (dependent variable):

X = data[['feature1', 'feature2', 'feature3']]  # Replace with actual feature names
y = data['price']  # The target variable, replace with actual column name

Step 4: Split the data into training and testing sets

We split the data into training and testing sets to avoid overfitting:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 5: Train the model

model = LinearRegression()
model.fit(X_train, y_train)

Step 6: Make predictions and evaluate

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f'Mean Squared Error: {mse}')
print(f'R-squared: {r2}')

Regularization Techniques: Ridge (L2) and Lasso (L1) Regression

Linear regression assumes a linear relationship, and without constraints, it can sometimes overfit, especially when there are many features. Regularization techniques like Ridge and Lasso help to prevent overfitting by adding a penalty to the coefficients.

Ridge Regression (L2 Regularization)

Ridge regression adds a penalty equal to the sum of the squared coefficients to the cost function:

$latex J(\beta) = \sum_{i=1}^{n} (y_i – \hat{y}i)^2 + \lambda \sum{j=1}^{n} \beta_j^2$

The $latex \lambda$ parameter controls the amount of regularization. A higher $latex \lambda$ shrinks the coefficients towards zero.

Lasso Regression (L1 Regularization)

Lasso regression adds a penalty equal to the absolute value of the coefficients:

$latex J(\beta) = \sum_{i=1}^{n} (y_i – \hat{y}i)^2 + \lambda \sum{j=1}^{n} |\beta_j|$

Lasso can shrink some coefficients to zero, effectively selecting important features and performing variable selection.

Evaluation Metrics for Linear Regression

Evaluating a regression model requires understanding how well the model predicts the target variable. Two common evaluation metrics are:

Mean Squared Error (MSE)

MSE measures the average squared difference between the actual and predicted values:

$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

A lower MSE indicates a better model fit.

R-squared (Coefficient of Determination)

R-squared represents the proportion of variance in the dependent variable explained by the independent variables:

$R^2 = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}}$

An $R^2$ close to 1 means the model fits the data well.

0 0 votes

Article Rating

Linear Regression

Table of Contents