Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better.
Now lets address your second query lets look into what is one-hot encoding and dummy encoding and then see the difference
One hot Encoding: Take the example of column name Fruit which can have different types of fruits like Blackberry, Grape, Orange. Here each category is mapped to binary variable containing either 0 or 1. Widely utilized when features are nominal.
Fruit Price (dollars per pound) Blackberry 3.82 Grape 1.2 Orange .64 Post one hot encoding the table now looks as shown below
One Hot Encoded table
Blackberry Grape Orange Price (dollars per pound) 1 0 0 3.82 0 1 0 1.2 0 0 1 .64 Dummy Encoding: similar to one hot encoding. While one hot encoding utilises
Nbinary variables forNcategories in a variable. Dummy encoding usesN-1features to representNlabels/categoriesOne Hot Coding Vs Dummy Coding
Column One Hot Code Dummy Code Blackberry 100 10 Grape 010 01 Orange 001 00
Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better.
Now lets address your second query lets look into what is one-hot encoding and dummy encoding and then see the difference
One hot Encoding: Take the example of column name Fruit which can have different types of fruits like Blackberry, Grape, Orange. Here each category is mapped to binary variable containing either 0 or 1. Widely utilized when features are nominal.
Fruit Price (dollars per pound) Blackberry 3.82 Grape 1.2 Orange .64 Post one hot encoding the table now looks as shown below
One Hot Encoded table
Blackberry Grape Orange Price (dollars per pound) 1 0 0 3.82 0 1 0 1.2 0 0 1 .64 Dummy Encoding: similar to one hot encoding. While one hot encoding utilises
Nbinary variables forNcategories in a variable. Dummy encoding usesN-1features to representNlabels/categoriesOne Hot Coding Vs Dummy Coding
Column One Hot Code Dummy Code Blackberry 100 10 Grape 010 01 Orange 001 00
The purpose of one-hot encoding is to assign numbers to categorical variables which does not create a false, meaningless numerical pattern.
If you have categorical variables "Apple", "Orange", "Cherry", "Tomato" and you assign them numerical values 0, 1, 2, 3, then these numerical values have interpretations like "Cherry is between Tomato and Apple, but closer to Tomato" because 2 is between 0 and 3, but closer to 3. This is nonsense. It's bad nonsense, because algorithms to analyze this data (like regressions, or whatever) can pick up on it and read too much into it.
If you instead represent "Apple", "Orange", "Cherry", and "Tomato" as the 4-tuples (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1), then you don't have this problem. Each coordinate is either 0 or 1, and measures the "Appleness" or the "Orangeness" or the "Cherriness" or the "Tomatoness" of your fruit. That's one-hot encoding.
As an example of this, suppose that the average apple weighs 200 grams, the average orange weighs 150 grams, the average cherry 30 grams, and the average tomato 100 grams. With one-hot encoding 
, this average weight is a linear function of the encoding: $200x_1 + 150x_2 + 30x_3 + 100x_4$. This is something a regression can figure out from data. With the 0, 1, 2, 3 encoding, there's no nice function that will give you the average weight of a fruit given its number.
Now, as for dummy encoding: one-hot encoding still has a problem, which is that the linear function is not unique. The function $100 + 100x_1 + 50x_2 - 70x_3$ gives the same values as the previous function at the points (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1). That's because each valid point 
satisfies 
.
(Again, this is not just a curiosity; this affects the way we analyze the data. For example, linear regressions behave badly when an 
-dimensional input doesn't actually range freely across all 
dimensions.)
Dummy encoding drops one of the coordinates, since it can be inferred from the other three, to avoid this issue. The four fruits might be encoded as (1,0,0), (0,1,0), (0,0,1), and (0,0,0).
python - What's the difference between dummy variable and one-hot encoding? - Stack Overflow
regression - One-hot vs dummy encoding in Scikit-learn - Cross Validated
What's the difference between one hot encoding and dummy encoding?
regression - Problems with one-hot encoding vs. dummy encoding - Cross Validated
In fact, there is no difference in the effect of the two approaches (rather wordings) on your regression.
In either case, you have to make sure that one of your dummies is left out (i.e. serves as base assumption) to avoid perfect multicollinearity among the set.
For instance, if you want to take the weekday of an observation into account, you only use 6 (not 7) dummies assuming the one left out to be the base variable. When using one-hot encoding, your weekday variable is present as a categorical value in one single column, effectively having the regression use the first of its values as the base.
Technically 6- a day week is enough to provide a unique mapping for a vocabulary of size 7:
1. Sunday [0,0,0,0,0,0]
2. Monday [1,0,0,0,0,0]
3. Tuesday [0,1,0,0,0,0]
4. Wednesday [0,0,1,0,0,0]
5. Thursday [0,0,0,1,0,0]
6. Friday [0,0,0,0,1,0]
7. Saturday [0,0,0,0,0,1]
dummy coding is a more compact representation, it is preferred in statistical models that perform better when the inputs are linearly independent.
Modern machine learning algorithms, though, don’t require their inputs to be linearly independent and use methods such as L1 regularization to prune redundant inputs. The additional degree of freedom allows the framework to transparently handle a missing input in production as all zeros.
1. Sunday [0,0,0,0,0,0,1]
2. Monday [0,0,0,0,0,1,0]
3. Tuesday [0,0,0,0,1,0,0]
4. Wednesday [0,0,0,1,0,0,0]
5. Thursday [0,0,1,0,0,0,0]
6. Friday [0,1,0,0,0,0,0]
7. Saturday [1,0,0,0,0,0,0]
for missing values : [0,0,0,0,0,0,0]
Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to True? I do not see any "warning" on the website.
For an unregularized linear model with one-hot encoding, yes, you need to set the intercept to be false or else incur perfect collinearity. sklearn also allows for a ridge shrinkage penalty, and in that case it is not necessary, and in fact you should include both the intercept and all the levels. For dummy encoding you should include an intercept, unless you have standardized all your variables, in which case the intercept is zero.
Since one-hot encoding generates more variables, does it have more degree of freedom than dummy encoding?
The intercept is an additional degree of freedom, so in a well specified model it all equals out.
For the second one, what if there are k categorical variables? k variables are removed in dummy encoding. Is the degree of freedom still the same?
You could not fit a model in which you used all the levels of both categorical variables, intercept or not. For, as soon as you have one-hot-encoded all the levels in one variable in the model, say with binary variables 
, then you have a linear combination of predictors equal to the constant vector
If you then try to enter all the levels of another categorical 
into the model, you end up with a distinct linear combination equal to a constant vector
and so you have created a linear dependency
So you must leave out a level in the second variable, and everything lines up properly.
Say, I have 3 categorical variables, each of which has 4 levels. In dummy encoding, 3*4-3=9 variables are built with one intercept. In one-hot encoding, 3*4=12 variables are built without an intercept. Am I correct?
The second thing does not actually work. The 
column design matrix you create will be singular. You need to remove three columns, one from each of three distinct categorical encodings, to recover non-singularity of your design.
To add a little to @MatthewDrury's answer regarding this question:
Say, I have 3 categorical variables, each of which has 4 levels. In dummy encoding, 3*4-3=9 variables are built with one intercept. In one-hot encoding, 3*4=12 variables are built without an intercept. Am I correct?
We can examine what the design matrix would look like with and without an intercept by using model.matrix from R.
With an intercept:
> df <- expand.grid(w = letters[1:4], x = letters[5:8], y = letters[9:12])
> model.matrix(~ w + x + y, df)
(Intercept) wb wc wd xf xg xh yj yk yl
1 1 0 0 0 0 0 0 0 0 0
2 1 1 0 0 0 0 0 0 0 0
3 1 0 1 0 0 0 0 0 0 0
4 1 0 0 1 0 0 0 0 0 0
5 1 0 0 0 1 0 0 0 0 0
6 1 1 0 0 1 0 0 0 0 0
7 1 0 1 0 1 0 0 0 0 0
8 1 0 0 1 1 0 0 0 0 0
9 1 0 0 0 0 1 0 0 0 0
10 1 1 0 0 0 1 0 0 0 0
11 1 0 1 0 0 1 0 0 0 0
12 1 0 0 1 0 1 0 0 0 0
13 1 0 0 0 0 0 1 0 0 0
14 1 1 0 0 0 0 1 0 0 0
15 1 0 1 0 0 0 1 0 0 0
16 1 0 0 1 0 0 1 0 0 0
17 1 0 0 0 0 0 0 1 0 0
18 1 1 0 0 0 0 0 1 0 0
19 1 0 1 0 0 0 0 1 0 0
20 1 0 0 1 0 0 0 1 0 0
21 1 0 0 0 1 0 0 1 0 0
22 1 1 0 0 1 0 0 1 0 0
23 1 0 1 0 1 0 0 1 0 0
24 1 0 0 1 1 0 0 1 0 0
25 1 0 0 0 0 1 0 1 0 0
26 1 1 0 0 0 1 0 1 0 0
27 1 0 1 0 0 1 0 1 0 0
28 1 0 0 1 0 1 0 1 0 0
29 1 0 0 0 0 0 1 1 0 0
30 1 1 0 0 0 0 1 1 0 0
31 1 0 1 0 0 0 1 1 0 0
32 1 0 0 1 0 0 1 1 0 0
33 1 0 0 0 0 0 0 0 1 0
34 1 1 0 0 0 0 0 0 1 0
35 1 0 1 0 0 0 0 0 1 0
36 1 0 0 1 0 0 0 0 1 0
37 1 0 0 0 1 0 0 0 1 0
38 1 1 0 0 1 0 0 0 1 0
39 1 0 1 0 1 0 0 0 1 0
40 1 0 0 1 1 0 0 0 1 0
41 1 0 0 0 0 1 0 0 1 0
42 1 1 0 0 0 1 0 0 1 0
43 1 0 1 0 0 1 0 0 1 0
44 1 0 0 1 0 1 0 0 1 0
45 1 0 0 0 0 0 1 0 1 0
46 1 1 0 0 0 0 1 0 1 0
47 1 0 1 0 0 0 1 0 1 0
48 1 0 0 1 0 0 1 0 1 0
49 1 0 0 0 0 0 0 0 0 1
50 1 1 0 0 0 0 0 0 0 1
51 1 0 1 0 0 0 0 0 0 1
52 1 0 0 1 0 0 0 0 0 1
53 1 0 0 0 1 0 0 0 0 1
54 1 1 0 0 1 0 0 0 0 1
55 1 0 1 0 1 0 0 0 0 1
56 1 0 0 1 1 0 0 0 0 1
57 1 0 0 0 0 1 0 0 0 1
58 1 1 0 0 0 1 0 0 0 1
59 1 0 1 0 0 1 0 0 0 1
60 1 0 0 1 0 1 0 0 0 1
61 1 0 0 0 0 0 1 0 0 1
62 1 1 0 0 0 0 1 0 0 1
63 1 0 1 0 0 0 1 0 0 1
64 1 0 0 1 0 0 1 0 0 1
Without an intercept:
> model.matrix(~ w + x + y - 1, df)
wa wb wc wd xf xg xh yj yk yl
1 1 0 0 0 0 0 0 0 0 0
2 0 1 0 0 0 0 0 0 0 0
3 0 0 1 0 0 0 0 0 0 0
4 0 0 0 1 0 0 0 0 0 0
5 1 0 0 0 1 0 0 0 0 0
6 0 1 0 0 1 0 0 0 0 0
7 0 0 1 0 1 0 0 0 0 0
8 0 0 0 1 1 0 0 0 0 0
9 1 0 0 0 0 1 0 0 0 0
10 0 1 0 0 0 1 0 0 0 0
11 0 0 1 0 0 1 0 0 0 0
12 0 0 0 1 0 1 0 0 0 0
13 1 0 0 0 0 0 1 0 0 0
14 0 1 0 0 0 0 1 0 0 0
15 0 0 1 0 0 0 1 0 0 0
16 0 0 0 1 0 0 1 0 0 0
17 1 0 0 0 0 0 0 1 0 0
18 0 1 0 0 0 0 0 1 0 0
19 0 0 1 0 0 0 0 1 0 0
20 0 0 0 1 0 0 0 1 0 0
21 1 0 0 0 1 0 0 1 0 0
22 0 1 0 0 1 0 0 1 0 0
23 0 0 1 0 1 0 0 1 0 0
24 0 0 0 1 1 0 0 1 0 0
25 1 0 0 0 0 1 0 1 0 0
26 0 1 0 0 0 1 0 1 0 0
27 0 0 1 0 0 1 0 1 0 0
28 0 0 0 1 0 1 0 1 0 0
29 1 0 0 0 0 0 1 1 0 0
30 0 1 0 0 0 0 1 1 0 0
31 0 0 1 0 0 0 1 1 0 0
32 0 0 0 1 0 0 1 1 0 0
33 1 0 0 0 0 0 0 0 1 0
34 0 1 0 0 0 0 0 0 1 0
35 0 0 1 0 0 0 0 0 1 0
36 0 0 0 1 0 0 0 0 1 0
37 1 0 0 0 1 0 0 0 1 0
38 0 1 0 0 1 0 0 0 1 0
39 0 0 1 0 1 0 0 0 1 0
40 0 0 0 1 1 0 0 0 1 0
41 1 0 0 0 0 1 0 0 1 0
42 0 1 0 0 0 1 0 0 1 0
43 0 0 1 0 0 1 0 0 1 0
44 0 0 0 1 0 1 0 0 1 0
45 1 0 0 0 0 0 1 0 1 0
46 0 1 0 0 0 0 1 0 1 0
47 0 0 1 0 0 0 1 0 1 0
48 0 0 0 1 0 0 1 0 1 0
49 1 0 0 0 0 0 0 0 0 1
50 0 1 0 0 0 0 0 0 0 1
51 0 0 1 0 0 0 0 0 0 1
52 0 0 0 1 0 0 0 0 0 1
53 1 0 0 0 1 0 0 0 0 1
54 0 1 0 0 1 0 0 0 0 1
55 0 0 1 0 1 0 0 0 0 1
56 0 0 0 1 1 0 0 0 0 1
57 1 0 0 0 0 1 0 0 0 1
58 0 1 0 0 0 1 0 0 0 1
59 0 0 1 0 0 1 0 0 0 1
60 0 0 0 1 0 1 0 0 0 1
61 1 0 0 0 0 0 1 0 0 1
62 0 1 0 0 0 0 1 0 0 1
63 0 0 1 0 0 0 1 0 0 1
64 0 0 0 1 0 0 1 0 0 1
We can see that when we use an intercept, model.matrix uses dummy encoding with each variable w, x, and y being turned into 3 dummy variables, plus an intercept column. So there is a total of 10 degrees of freedom.
When we don't use an intercept, model.matrix creates 4 dummy variables for w and 3 dummy variables for x and y (and no intercept column). So the number of degrees of freedom is still 10.
Hi everyone,
There are two ways to encode categorical variables. One hot encoding and dummy encoding. I'm studied statistics and we always used dummy encoding because we don't want to see multicollinearity (dummy variable trap). Now I'm following the Kaggle competitions and everyone is using one hot enc. I'm not a "traditional" statistician (so I don't say dummy is the true one or something like that) but I'm just trying to understand why.
thanks in advance!
The issue with representing a categorical variable that has $k$ levels with $k$ variables in regression is that, if the model also has a constant term, then the terms will be linearly dependent and hence the model will be unidentifiable. For example, if the model is $μ = a_0 + a_1X_1 + a_2X_2$ and $X_2 = 1 - X_1$, then any choice $(β_0, β_1, β_2)$ of the parameter vector is indistinguishable from $(β_0 + β_2,\; β_1 - β_2,\; 0)$. So although software may be willing to give you estimates for these parameters, they aren't uniquely determined and hence probably won't be very useful.
Penalization will make the model identifiable, but redundant coding will still affect the parameter values in weird ways, given the above.
The effect of a redundant coding on a decision tree (or ensemble of trees) will likely be to overweight the feature in question relative to others, since it's represented with an extra redundant variable and therefore will be chosen more often than it otherwise would be for splits.
I feel the best answer to this question is buried in the comments by @MatthewDrury, which states that there is a difference and that you should use the seemingly redundant column in any regularized approach. @MatthewDrury's reasoning is
[In regularized regression], the intercept is not penalized, so if you are inferring the effect of a level as not part of the intercept, its hard to say you are penalizing all levels equally. Instead, always include all the levels, so each is symmetric with respect to the penalty.
I think he's got a point.
Most clustering algorithms will be distance based.
Any such encoding is a hack to make categoricial data look as if it were numeric, but this only postpones the resulting problems: how to normalize, weight, decorrelate, and combine features.
For most clustering algorithms, it makes an enormous difference whether you dummy encode as 0,1 or as 0,100000 or as 0,0.000001. So which one should you use? There is no objective mathematical answer to this, and it causes severe problems.
The main difference is that dummy encoding typically removes one of the columns. E.g. a variable with 3 levels will get 2 dummy variables and 3 one-hot-encoded variables. This is to ensure that you have no multicollinearity. One-hot-encoding is sometimes also called full dummy encoding