Communities for your favorite technologies. Explore all Collectives · Stack Overflow for Teams is now called Stack Internal. Bring the best of human thought and AI automation together at your work More on stackoverflow.com

Regularization doesn't necessarily have anything to do with improving your accuracy or minimizing your loss. Rather, regularization is generally effective in situations where the coefficients exhibit high variance, since these cases are normally dominated by the noise of the data. Regularization allows us to bias our model towards "rules of thumb" rather than blindly following statistical noise, which can be desirable in some contexts depending on the goals. High variance in the coefficients usually comes from either (1) very noisy data, or (2) correlation between variables. In either case, the loss landscape of the coefficients exhibits a very "broad" minima, instead of a "sharp" minima which is more stable under small perturbations. Regularization is a way of "sharpening" the broad minima, injecting stability through bias. Instead of letting the exact minimum value be dominated by random error (which is the case in high-variance contexts), we bias the value according to what basically amounts as rules-of-thumb. L1 regularization biases towards sparsity, selecting the smallest set of coefficients that achieve similar accuracy (and selecting the smallest-valued coefficients among sets of equal size). L2 regularization is the opposite, "spreading out" the predictive weights among the coefficients as much as possible. Illustrative simple example using linear regression: suppose X1 and X2 are highly correlated in a 2:1 ratio. If the true model is y = 3X1 + 2X2, this will look a whole lot like (X1 + 6X2), or (2X1 + 4X2), etc, allowing basically all coefficient sets that satisfy 2*B1 + B2 = 8 (depending on the degree of collinearity). This is because the high degree of collinearity creates a "taco" shape in the loss landscape, relating model loss as a function of B1 and B2. In this example L1 regularization would end up choosing y = 4X1 (retaining the smallest coefficient of the correlated set), while L2 regularization would end up choosing y = 8/3 X1 + 8/3 X2 (balancing B1 and B2 values). More on reddit.com

L1 regularization helps perform feature selection in sparse feature spaces, and that is a good practical reason to use L1 in some situations. However, beyond that particular reason I have never seen L1 to perform better than L2 in practice. If you take a look at LIBLINEAR FAQ on this issue you will see how they have not seen a practical example where L1 beats L2 and encourage users of the library to contact them if they find one. Even in a situation where you might benefit from L1's sparsity in order to do feature selection, using L2 on the remaining variables is likely to give better results than L1 by itself. More on reddit.com

I’m a big fan of the Bayesian interpretation of L1 and L2 regularization. Under a Bayesian paradigm, L1 regularization is equal to a Laplace (double exponential) prior on the parameter. Intuitively, you can think about this as saying, “I really think this parameter should be zero and unless we have compelling evidence otherwise it should stay at zero”. This is reflected by the pdf of the prior, with a laplace prior we have a really sharp pointy part centered right at 0 that exponentially falls off as we move away from 0. A Laplace prior is considered sparsity inducing, which means lots of our parameters will end up being 0. Compare this to L2 regularization which is equivalent to a Gaussian prior mean centered at 0. We’re still expecting the parameters to be close to 0, but we don’t have nearly as strong as assumption as we do when we use a Laplace prior. We essentially allow the parameters to have a little more wiggle room around 0, which leads to lots of our parameters being close to, but not actually 0. More on reddit.com