Linear regression is one of the foundational techniques in machine learning, widely used for predictive modelling. It has numerous applications in fields like economics, biology, and engineering. While the basic linear regression is straightforward, more advanced regularisation techniques and feature transformations, such as Ridge regression, Lasso regression, ElasticNet, and Polynomial features, significantly enhance its capabilities. This post delves into the theory of these methods, discusses their mathematical underpinnings, and walks through their Python implementations using the provided scripts and visual examples. We will explore not only the implementation but also the deeper theoretical aspects to provide a thorough understanding. > The complete code for this blog post is available on [GitHub](https://github.com/TristanBester/Machine-Learning-From-The-Ground-Up).

## **Linear Regression: A Mathematical Perspective** Linear regression models the relationship between a dependent variable $y$ and one or more independent variables $X$ by fitting a linear equation: $$ \hat{y} = X \beta + \epsilon, $$ where: - $\hat{y}$ is the predicted response, - $X$ is the design matrix (containing the features), - $\beta$ is the vector of coefficients, and - $\epsilon$ is the residual error. The coefficients $\beta$ are obtained by minimising the residual sum of squares (RSS): $$ RSS = \| y - X \beta \|_2^2. $$ This minimisation problem has a closed-form solution: $$ \beta = (X^T X)^{-1} X^T y. $$ ### Theoretical Insight The closed-form solution works under the assumption that $X^T X$ is invertible. If the matrix $X$ has highly correlated features, this assumption may not hold due to multicollinearity. Additionally, if the number of features exceeds the number of observations, $X^T X$ becomes singular. These limitations form the motivation for regularisation techniques discussed later. ### Python Implementation of Linear Regression The provided script `linear_model.py` implements linear regression from scratch. Below is an excerpt of the implementation: ```python import numpy as np class LinearRegression: def __init__(self): self.coefficients = None def fit(self, X, y): X = np.insert(X, 0, 1, axis=1) # Adding bias term self.coefficients = np.linalg.inv(X.T @ X) @ X.T @ y def predict(self, X): X = np.insert(X, 0, 1, axis=1) # Adding bias term return X @ self.coefficients ``` - **`fit` Method**: Implements the closed-form solution for $\beta$ using the normal equation. - **`predict` Method**: Uses the learned coefficients to make predictions for new data points. The example usage of this model is demonstrated in `linear_model_example.py`, which generates a synthetic dataset and visualises the results: ```python from linear_model import LinearRegression # Generate synthetic data X = np.random.rand(100, 1) * 10 y = 2 * X.squeeze() + 1 + np.random.randn(100) # Fit the model model = LinearRegression() model.fit(X, y) # Predict and visualise predictions = model.predict(X) ``` Below is an example plot generated by the script, illustrating the fitted line: Linear Regression Fit

## **Ridge Regression** Linear regression can suffer from overfitting, especially when $X$ is highly correlated or when the number of features is large relative to the number of observations. Ridge regression addresses this issue by adding a penalty term to the RSS: $$ RSS_{ridge} = \| y - X \beta \|_2^2 + \lambda \| \beta \|_2^2. $$ Here, $\lambda > 0$ is the regularisation parameter controlling the trade-off between minimising the RSS and shrinking the coefficients. ### Derivation of the Ridge Solution The modified objective function can be solved analytically by taking its derivative with respect to $\beta$ and setting it to zero: $$ (X^T X + \lambda I) \beta = X^T y, $$ where $I$ is the identity matrix. Solving for $\beta$, we obtain: $$ \beta_{ridge} = (X^T X + \lambda I)^{-1} X^T y. $$ ### Impact of Regularisation The penalty encourages smaller coefficients, reducing model complexity and mitigating overfitting. Unlike standard linear regression, Ridge regression stabilises the solution when $X^T X$ is nearly singular.

## **Lasso Regression** While Ridge regression shrinks coefficients towards zero, it does not perform feature selection. Lasso regression introduces an $L_1$ penalty, which encourages sparsity in the coefficients: $$ RSS_{lasso} = \| y - X \beta \|_2^2 + \lambda \| \beta \|_1. $$ ### Theoretical Implications of $L_1$ Regularisation The $L_1$ norm penalty can force some coefficients to become exactly zero, effectively selecting a subset of features. This property is particularly useful in high-dimensional settings where feature selection is critical. Lasso regression does not have a closed-form solution and is typically solved using optimisation algorithms such as coordinate descent. Below is an example plot illustrating the regression coefficients:

## **ElasticNet Regression** ElasticNet combines Ridge and Lasso penalties to provide a balance between their strengths: $$ RSS_{elastic} = \| y - X \beta \|_2^2 + \lambda_1 \| \beta \|_1 + \lambda_2 \| \beta \|_2^2. $$ ### Motivation and Applications ElasticNet is particularly useful when features are highly correlated. While Lasso may arbitrarily select one feature from a group of correlated features, ElasticNet tends to include all relevant features, distributing the penalty among them. Below is an example plot illustrating the ElasticNet regularisation path:

## **Polynomial Features** Linear regression assumes a linear relationship between $X$ and $y$. To model non-linear relationships, we can transform the features using polynomial basis functions: $$ X_{poly} = \begin{bmatrix} 1 & x_1 & x_1^2 & \dots & x_1^d \\ 1 & x_2 & x_2^2 & \dots & x_2^d \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \dots & x_n^d \end{bmatrix}. $$ ### Python Implementation of Polynomial Features The script `polynomial_features.py` implements this transformation: ```python import numpy as np class PolynomialFeatures: def __init__(self, degree): self.degree = degree def transform(self, X): X_poly = np.ones((X.shape[0], 1)) for d in range(1, self.degree + 1): X_poly = np.hstack((X_poly, X ** d)) return X_poly ``` This transformation can be seamlessly integrated into the linear regression pipeline. For example, `polynomial_example.py` demonstrates the fitting of a polynomial regression model: ```python from polynomial_features import PolynomialFeatures from linear_model import LinearRegression # Generate data X = np.linspace(0, 10, 100).reshape(-1, 1) y = 1 + 2 * X + 3 * X**2 + np.random.randn(100, 1) # Transform features poly = PolynomialFeatures(degree=2) X_poly = poly.transform(X) # Fit model model = LinearRegression() model.fit(X_poly, y) ``` Below is an example plot illustrating polynomial regression: Polynomial Regression Fit

## **Conclusion** Linear regression forms the foundation of many machine learning models. However, its versatility is expanded through techniques like Ridge regression, Lasso regression, ElasticNet, and Polynomial features. Each method addresses specific challenges, such as overfitting, feature selection, and non-linear relationships. Ridge stabilises solutions, Lasso performs feature selection, ElasticNet balances between the two, and Polynomial transformations enable modelling complex relationships. By implementing these methods from scratch, as demonstrated above, you gain a deeper understanding of their mechanics, enabling you to apply them effectively in your machine learning projects.