statistical optimality criterion
least absolute deviations regression method diagram
Least absolute deviations (LAD), also known as least absolute errors (LAE), least absolute residuals (LAR), or least absolute values (LAV), is a statistical optimality criterion and a statistical optimization technique based on … Wikipedia
🌐
Wikipedia
en.wikipedia.org › wiki › Least_absolute_deviations
Least absolute deviations - Wikipedia
November 22, 2024 - Since it is known that at least one least absolute deviations line traverses at least two data points, this method will find a line by comparing the SAE (Smallest Absolute Error over data points) of each line, and choosing the line with the smallest SAE. In addition, if multiple lines have ...
Top answer
1 of 2
6

You want an example for solving least absolute deviation by linear programming. I will show you an simple implementation in R. Quantile regression is a generalization of least absolute deviation, which is the case of the quantile 0.5, so I will show a solution for quantile regression. Then you can check the results with the R quantreg package:

    rq_LP  <-  function(x, Y, r=0.5, intercept=TRUE) {
        require("lpSolve")
        if (intercept) X  <-  cbind(1, x) else X <-  cbind(x)
        N   <-  length(Y)
        n  <-  nrow(X)
        stopifnot(n == N)
        p  <-  ncol(X)
        c  <-  c(rep(r, n), rep(1-r, n), rep(0, 2*p))  
                 # cost coefficient vector
        A  <- cbind(diag(n), -diag(n), X, -X)
        res  <-  lp("min", c, A, "=", Y, compute.sens=1)
            ### Desempaquetar los coefs:
        sol <- res$solution
        coef1  <-  sol[(2*n+1):(2*n+2*p)]
        coef <- numeric(length=p)
        for (i in seq(along=coef)) {
             coef[i] <- (if(coef1[i]<=0)-1 else +1) *  
                  max(coef1[i], coef1[i+p])
        }
        return(coef)
        }

Then we use it in a simple example:

    library(robustbase)
    data(starsCYG)
    Y  <- starsCYG[, 2]
    x  <- starsCYG[, 1]
    rq_LP(x, Y)
    [1]  8.1492045 -0.6931818

then you yourself can do the check with quantreg.

2 of 2
2

Linear Programming can be generalized with convex optimization, where in addition to simplex, many more reliable algorithms are available.

I would suggest you to check The Convex Optimization Book and the CVX toolbox they provided. Where you can easily formulate least absolute deviation with regularization.

https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf

http://cvxr.com/cvx/

🌐
R-project
roi.r-forge.r-project.org › use_case_LAD.html
Least absolute deviation (LAD) problem
\[ \begin{eqnarray*} \underset{{\beta_0,\mathbf{\beta},\mathbf{e}^+,\mathbf{e}^-}}{\text{minimize}} ~~ \sum_{i=1}^n e_i^+ + e_i^- ~~~~~~~~~~~~~~~~~ \nonumber \\ \text{subject to} ~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~ \nonumber \\ \beta_0 + \mathbf{\beta}^\top \mathbf{x}_i + e_i^+ - e_i^- = 0 ~~~~~~ i = 1,\ldots{},n \nonumber \\ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \beta_j = -1 ~~~~~~~~~~~~~~~~~~~~~ \nonumber \\ ~~~~~~~~~~~~~~~~~~~~~~~ e_i^+, e_i^- \geq 0 ~~~~~ i = 1,\ldots{},n \end{eqnarray*} \] given a set of points \(\mathbf{x}_i \in \mathbb{R}^m\), \(i = 1,\ldots{},n\) and the \(j^{th}\) column representing the dependent variable \(y\). In comparison to the original formulation we differentiate between positive and negative deviations/errors (\(e_i^+\) and \(e_i^-\), respectively) to get to the linear programming problem shown above.
🌐
GitHub
github.com › seunghwanyoo › lad_reformulation
GitHub - seunghwanyoo/lad_reformulation: Least absolute deviation (LAD) problem with linear programming
Least absolute deviation (LAD) method is using L1 norm to get the solution x for Ax = b instead of L2 norm. We test a simple linear regression problem with LAD and compare its performance with Least Squares (LS) method.
Author   seunghwanyoo
🌐
Princeton
vanderbei.princeton.edu › 542 › lectures › lec9.pdf pdf
Linear Programming: Chapter 12 Regression Robert J. Vanderbei October 17, 2007
Least Absolute Deviation Regression via Lin- ear Programming · min · X · i · bi − · X · j · aijxj · Equivalent Linear Program: min · X · i · ti · −ti ≤bi − · X · j · aijxj ≤ti · i = 1, 2, . . . , m · AMPL Model ·
🌐
Reddit
reddit.com › r/learnmath › linear programming: how to find the least absolute deviation?
r/learnmath on Reddit: Linear Programming: How to find the Least Absolute Deviation?
July 5, 2023 -

I have been making a program to analyse sets of data. I have implemented linear regression via least squares, but I was hoping to implement Least Absolute Deviation. For clarity, LAD wishes to minimise the sum of | y_i - a - b x_i |. To set up the problem as a set of linear constraints, a new variable, e_i, is used. Setting e_i = | y_i - a - b x_i |, we can form the inequalities e_i >= y_i - a - b x_i and -e_i <= y_i - a - b x_i . I have implemented simplex method in my program, so I wished to get it into a form usable by the simplex method. As such, I have converted each to [variables] <= value form, being -y_i = -a - b x_i - e_i + s_{2i} and y_i = a + b x_i - e_i + s_{2i +1}. The objective function is z = sum_{i=1}^{n} e_i. Since we wish to minimise this, I have negated everything to give -z + sum_{i=1}^{n} e_i = 0. Is this the correct setup for use of simplex? That is to ask: if I were to form a tableau from the following and apply simplex, would I get the desired a, b such that y = b x + a is the LAD line?

Any help would be appreciated.

EDIT: This formulation doesn't seem to function since the only selectable pivot column is the "z" column which is all zeros. How can this be avoided?

🌐
Stack Overflow
stackoverflow.com › questions › 64422417 › irls-vs-linear-programming-for-large-scale-for-least-absolute-deviation-lad-r
IRLS vs. Linear Programming for Large Scale for Least Absolute Deviation (LAD) Regression - Stack Overflow
The cause for the speed is that the IRLS most consuming operation, solving a linear system, is done on A' * A which in your case is pretty small. ... Least Absolute Deviation (LAD) Line Fitting / Regression.
🌐
ResearchGate
researchgate.net › publication › 229703239_Least_squares_versus_minimum_absolute_deviation_estimation_in_linear_models
Least squares versus minimum absolute deviation estimation in linear models
June 7, 2007 - In particular, if the linear regression ... to a p equation linear programming model with bounded variables; and fitting by the Chebyshev criterion is exhibited to lead to a standard-form p+1 equation linear programming mode...
🌐
ResearchGate
researchgate.net › publication › 24112751_Least_Absolute_Deviation_Estimation_of_Linear_Econometric_Models_A_Literature_Review
(PDF) Least Absolute Deviation Estimation of Linear Econometric Models: A Literature Review
July 5, 2022 - Dantzig, GB (1963). Linear Programming and Extensions. Princeton University · Press, Princeton, N. J. Dasgupta, M (2004). Least Absolute Deviation Estimation of Multi-Equation Linear
Find elsewhere
🌐
Taylor & Francis
taylorandfrancis.com › knowledge › Engineering_and_technology › Engineering_support_and_special_topics › Least_absolute_deviations
Least absolute deviations - Knowledge and References | Taylor & Francis
L1-norm error, also known as least absolute deviations (LAD), least absolute errors (LAE) is basically minimizing the sum of the absolute differences (M) between the target value (Yi) and the estimated values (f(xi)).
🌐
Mobook
mobook.github.io › MO-book › notebooks › 02 › 02-lad-regression.html
2.2 Least Absolute Deviation (LAD) Regression — Companion code for the book "Hands-On Mathematical Optimization with Python"
It remains a cornerstone of modern ... introduces an alternative approach to traditional linear regression, employing linear optimization to optimize based on the Least Absolute Deviation (LAD) metric....
🌐
Blogger
yetanothermathprogrammingconsultant.blogspot.com › 2017 › 11 › lp-and-lad-regression.html
Yet Another Math Programming Consultant: Linear Programming and LAD Regression
November 9, 2017 - I believe any book on linear programming will mention LAD (Least Absolute Deviation) or \(\ell_1\) regression: minimize the sum of the absolute values of the residuals.
🌐
Munich Personal RePEc Archive
mpra.ub.uni-muenchen.de › 1781 › 1 › MPRA_paper_1781.pdf pdf
Least absolute deviation estimation of linear econometric ...
However, models with the disturbances ... that in such cases estimation by the Least Absolute Deviation (LAD) method performs well. This paper is an attempt to survey the literature on LAD estimation of single as well as multi-equation linear econometric models....
🌐
Springer
link.springer.com › home › book
Least Absolute Deviations: Theory, Applications and Algorithms | Springer Nature Link
LAD estimates resist undue effects from a feyv, large errors. Therefore. in addition to being robust, they also make good starting points for other iterative, robust procedures. The LAD criterion has great utility.
Authors   Peter BloomfieldWilliam L. Steiger
Pages   14
🌐
Readthedocs
gurobi-optimods.readthedocs.io › en › stable › mods › lad-regression.html
Least Absolute Deviation Regression - gurobi-optimods documentation v3.0.0
The distinction between this Mod and the Ordinary Least Squares regression algorithm from scikit-learn is the loss function. LADRegression chooses coefficients \(w\) of a linear model \(y = Xw\) so as to minimize the sum of absolute errors on a training dataset \((X, y)\). In other words, it aims to minimize the following loss function: ... The fitting algorithm of the LAD regression Mod is implemented by formulating the loss function as a Linear Program (LP), which is then solved using Gurobi.
🌐
Hong Kong University of Science and Technology
math.hkust.edu.hk › ~makchen › Paper › LAD.pdf pdf
Analysis of least absolute deviation By KANI CHEN
(p −q1)-vector. It is common to consider linear alternative hypothesis · H1 : β ∈Ω1 \ Ω0. Letting Ω1 = Rp gives the important special case of H1 : β /∈Ω0. ... shall develop a general approach to hypothesis testing under the least absolute deviation criterion.
Top answer
1 of 1
5

Let's formulate the Sum of Absolute Residuals / Least Absolute Deviation (LAD) / $ {L}_{1} $ Regression / Robust Regression:

$$ \arg \min_{x} {\left\| A x - b \right\|}_{1} $$

There are few options to solve this.

Iteratively Reweighted Least Squares (IRLS)

The main trick in the IRLS approach is that:

$$ {\left\| \boldsymbol{x} \right\|}_{p}^{p} = \sum_{i} {\left| {x}_{i} \right|}^{p - 2} {\left| {x}_{i} \right|}^{2} = \sum_{i} {w}_{i}^{2} {\left| {x}_{i} \right|}^{2} $$

On the other hand, it is known (Weighted Least Squares):

$$ \sum_{i = 1}^{m} {w}_{i} {\left| {A}_{i, :} x - {b}_{i} \right|}^{2} = {\left\| {W}^{\frac{1}{2}} \left( A x - b \right) \right\|}_{2}^{2} $$

The solution is given by:

$$ \arg \min_{x} {\left\| {W}^{\frac{1}{2}} \left( A x - b \right) \right\|}_{2}^{2} = {\left( {A}^{T} W A \right)}^{-1} {A}^{T} W b $$

In the case above, where $ p = 1 $, one could set $ {w}_{i} = {\left| {A}_{i, :} x - {b}_{i} \right|}^{-1} $.
Since the solution depends on $ x $ one could set iterative procedure:

$$ {x}^{k + 1} = {\left( {A}^{T} {W}^{k} A \right)}^{-1} {A}^{T} {W}^{k} b $$

Where $ {W}_{ii}^{k} = {w}_{i}^{k} = {\left| {A}_{i, :} {x}^{k} - {b}_{i} \right|}^{p - 2} = {\left| {A}_{i, :} {x}^{k} - {b}_{i} \right|}^{-1} $.

Sub Gradient Method

The Sub Gradient is given by:

$$ \frac{\partial {\left\| A x - b \right\|}_{1}}{\partial x} = {A}^{T} \operatorname{sgn} \left( A x - b \right) $$

So the solution is given by iterating:

$$ {x}^{k + 1} = {x}^{k} - \alpha \left[ {A}^{T} \operatorname{sgn} \left( A x - b \right) \right] $$

Linear Programming Conversion

This approach convert the problem into Linear Programming problem.

By defining $ {t}_{i} = \left| {A}_{i, :} x - {b}_{i} \right| $ the optimization problem becomes:

$$\begin{align*} \text{minimize} \quad & \boldsymbol{1}^{T} \boldsymbol{t} \\ \text{subject to} \quad & -\boldsymbol{t} \preceq A x -b \preceq \boldsymbol{t} \\ & \boldsymbol{0} \preceq \boldsymbol{t} \end{align*}$$

Which is a Linear Programming form of the problem.

MATLAB Code

I implemented and validated the approaches by comparing it to CVX solution. The MATLAB code is available at my Mathematics StackExchange Question 2603548 Repository.

Remark: Pay attention that the MATLAB code is a vanilla implementation. Real world implementation should take into account numerical issues (Like the inverse of a small number).

🌐
Quora
quora.com › How-do-you-solve-least-absolute-deviation-by-simplex-method-regression-optimization-quantile-regression-linear-programming-least-absolute-deviations-statistics
How to solve least absolute deviation by simplex method (regression, optimization, quantile regression, linear programming, least absolute deviations, statistics) - Quora
Answer: Simplex methods are the mostly preferred and used to solve the least absolute deviations problem LAD applied to Regression In regression model multiple we find MSE Minimise the SUM of the sqares of residuals Giving added weight in MSE to large residuals and in method out liers add gra...
🌐
Northwestern
optimization.mccormick.northwestern.edu › index.php › Optimization_with_absolute_values
Optimization with absolute values - optimization
September 26, 2020 - To minimize the deviation, the problem is formulated in a basic form as: as the objective function, and linear constraints are · The nonlinearity in this form generates from the absolute value function. However, by substituting for , the problem can be transformed into a linear problem.
🌐
ResearchGate
researchgate.net › publication › 309690752_An_Alternative_Algorithm_and_R_Programming_Implementation_for_Least_Absolute_Deviation_Estimator_of_the_Linear_Regression_Models
(PDF) An Alternative Algorithm and R Programming Implementation for Least Absolute Deviation Estimator of the Linear Regression Models
August 7, 2025 - Linear models: Least squares and · alternatives (2nd ed.). New York, NY: Springer-Verlag, Inc. Rousseeuw, P. J. & Leroy, A. M. (1987). Robust regression and outlier ... Wolberg, J. (2006). Data analysis using the method of least squares. Berlin · Heidelberg: Springer-Verlag. ... OGUNDELE ET AL. ... OGUNDELE ET AL. ... ... In the traditional regression models based on the least absolute deviations, the optimal parameters of the model are calculated based on a linear optimization problem.