I'm almost positive that you're encountering memory issues because str.get_dummies returns an array full of 1s and 0s, of datatype np.int64. This is quite different from the behavior of pd.get_dummies, which returns an array of values of datatype uint8.

This appears to be a known issue. However, there's been no update, nor fix, for the past year. Checking out the source code for str.get_dummies will indeed confirm that it is returning np.int64.

An 8 bit integer will take up 1 byte of memory, while a 64 bit integer will take up 8 bytes. I'm hopeful that memory problems can be avoided by finding an alternative way to one-hot encode Col2 which ensures the output are all 8 bit integers.

Here was my approach, beginning with your example:

df = pd.DataFrame({'Col1': ['X', 'Y', 'X'],
                   'Col2': ['a,b,c', 'a,b', 'b,d']})
df

    Col1    Col2
0   X       a,b,c
1   Y       a,b
2   X       b,d

1. Since Col1 contains simple, non-delimited strings, we can easily one-hot encode it using pd.get_dummies:

df = pd.get_dummies(df, columns=['Col1'])
df

    Col2    Col1_X  Col1_Y
0   a,b,c        1       0
1   a,b          0       1
2   b,d          1       0

So far so good.

df['Col1_X'].values.dtype
dtype('uint8')

2. Let's get a list of all unique substrings contained inside the comma-delimited strings in Col2:

vals = list(df['Col2'].str.split(',').values)
vals = [i for l in vals for i in l]
vals = list(set(vals))
vals.sort()
vals

['a', 'b', 'c', 'd']

3. Now we can loop through the above list of values and use str.contains to create a new column for each value, such as 'a'. Each row in a new column will contain 1 if that row actually has the new column's value, such as 'a', inside its string in Col2. As we create each new column, we make sure to convert its datatype to uint8:

col='Col2'
for v in vals:
    n = col + '_' + v
    df[n] = df[col].str.contains(v)
    df[n] = df[n].astype('uint8')

df.drop(col, axis=1, inplace=True)
df

    Col1_X  Col1_Y  Col2_a  Col2_b  Col2_c  Col2_d
0        1       0       1       1       1       0
1        0       1       1       1       0       0
2        1       0       0       1       0       1

This results in a dataframe that meets your desired format. And thankfully, the integers in the four new columns that were one-hot encoded from Col2 only take up 1 byte each, as opposed to 8 bytes each.

df['Col2_a'].dtype
dtype('uint8')

If, on the outside chance, the above approach doesn't work. My advice would be to use str.get_dummies to one-hot encode Col2 in chunks of rows. Each time you do a chunk, you would convert its datatype from np.int64 to uint8, and then transform the chunk to a sparse matrix. You could eventually concatenate all chunks back together.

It's been a few years, so this may well not have been in the pandas toolkit back when this question was originally asked, but this approach seems a little easier to me. idxmax will return the index corresponding to the largest element (i.e. the one with a 1). We do axis=1 because we want the column name where the 1 occurs.

EDIT: I didn't bother making it categorical instead of just a string, but you can do that the same way as @Jeff did by wrapping it with pd.Categorical (and pd.Series, if desired).

In [1]: import pandas as pd

In [2]: s = pd.Series(['a', 'b', 'a', 'c'])

In [3]: s
Out[3]: 
0    a
1    b
2    a
3    c
dtype: object

In [4]: dummies = pd.get_dummies(s)

In [5]: dummies
Out[5]: 
   a  b  c
0  1  0  0
1  0  1  0
2  1  0  0
3  0  0  1

In [6]: s2 = dummies.idxmax(axis=1)

In [7]: s2
Out[7]: 
0    a
1    b
2    a
3    c
dtype: object

In [8]: (s2 == s).all()
Out[8]: True

EDIT in response to @piRSquared's comment: This solution does indeed assume there's one 1 per row. I think this is usually the format one has. pd.get_dummies can return rows that are all 0 if you have drop_first=True or if there are NaN values and dummy_na=False (default) (any cases I'm missing?). A row of all zeros will be treated as if it was an instance of the variable named in the first column (e.g. a in the example above).

If drop_first=True, you have no way to know from the dummies dataframe alone what the name of the "first" variable was, so that operation isn't invertible unless you keep extra information around; I'd recommend leaving drop_first=False (default).

Since dummy_na=False is the default, this could certainly cause problems. Please set dummy_na=True when you call pd.get_dummies if you want to use this solution to invert the "dummification" and your data contains any NaNs. Setting dummy_na=True will always add a "nan" column, even if that column is all 0s, so you probably don't want to set this unless you actually have NaNs. A nice approach might be to set dummies = pd.get_dummies(series, dummy_na=series.isnull().any()). What's also nice is that idxmax solution will correctly regenerate your NaNs (not just a string that says "nan").

It's also worth mentioning that setting drop_first=True and dummy_na=False means that NaNs become indistinguishable from an instance of the first variable, so this should be strongly discouraged if your dataset may contain any NaN values.

IIUC, you can use Pandas.Categorical dtype to handle this.

Example

# Setup
np.random.seed(0)
df = pd.DataFrame(np.random.choice(['A', 'B', 'C'], 6), columns=['cat'])
print(df)

[out]

  cat
0   A
1   B
2   A
3   B
4   B
5   C

And running pandas.get_dummies on this yields...

pd.get_dummies(df['cat'])

[out]

   A  B  C
0  1  0  0
1  0  1  0
2  1  0  0
3  0  1  0
4  0  1  0
5  0  0  1    

Now cast this Series to categorical dtype, and pass in the list of known categories...

categories = ['A', 'B', 'C', 'D', 'E']
df['cat'] = pd.Categorical(df['cat'], categories=categories)

pd.get_dummies(df['cat'])

[out]

   A  B  C  D  E
0  1  0  0  0  0
1  0  1  0  0  0
2  1  0  0  0  0
3  0  1  0  0  0
4  0  1  0  0  0
5  0  0  1  0  0

Nope. The column from priority or cluster won't be misinterpreted as the third column of severity.

Here's answer to how reference is kept:

in pandas.get_dummies there is a parameter i.e. drop_first allows you whether to keep or remove the reference (whether to keep k or k-1 dummies out of k categorical levels).

Please note drop_first = False meaning that the reference is not dropped and k dummies created out of k categorical levels! You set drop_first = True, then it will drop the reference column after encoding.

Here's link to one hot encoding.

As in your case severity has 3 categories S1, S2 and S3. After creating dummies one of these categories will always be 1 and others 0.

for s1 it will be [1,0,0], s2 will be [0,1,0] and s3 will be [0,0,1]

Now if you drop the column for category s1.

The values will be [0,0] if severity is S1

[1,0] if severity is S2

[0,1] if severity is S3.

So there is no information loss here and your model has one less column to deal with. That's why it is always recommended to keep drop_first parameter as True.

Edit :

After applying the dummies you will get columns like:

severity_S1   severity_S2   severity_S3  

  1              0              0                  # when value is S1
  0              1              0                  # when value is S2  
  0              0              1                  # when value is S3

pandas.get_dummies() drops the 1st column after creating the above references. So in your data will be like below:

 severity_S2   severity_S3

   0              0                  # when value is S1
   1              0                  # when value is S2  
   0              1                  # when value is S3

For all there variables your final data will look like below: I'm using short column names due to space issue:

s2  s3  p2  p3  B  C  D
0   0   1   0   1  0  0     # For row with S1, P2 and B
0   1   0   1   0  1  0     # For row with S3, P3 and C
1   0   0   0   0  0  1     # For row with S2, P1 and D
1   0   0   0   0  0  0     # For row with S2, P1 and A

Here are multiple ways you can do:

from sklearn.preprocessing import LabelEncoder

lbl=LabelEncoder()
df['Sex_encoded'] = lbl.fit_transform(df['Sex'])

# using only pandas
df['Sex_encoded'] = df['Sex'].map({'male': 0, 'female': 1})

   Survived  Pclass     Sex   Age     Fare  Sex_encoded
0         0       3    male  22.0   7.2500            0
1         1       1  female  38.0  71.2833            1
2         1       3  female  26.0   7.9250            1
3         1       1  female  35.0  53.1000            1
4         0       3    male  35.0   8.0500            0

categories = ['Sex', 'Embarked', 'Identity', 'Religion', ...]
sex, embark, identity, religion, ... = [pd.get_dummies(train[c], drop_first=True) for c in categories]

When I think of dummy variables I think of using them in the context of OLS regression, and I would do something like this:

import numpy as np
import pandas as pd
import statsmodels.api as sm

my_data = np.array([[5, 'a', 1],
                    [3, 'b', 3],
                    [1, 'b', 2],
                    [3, 'a', 1],
                    [4, 'b', 2],
                    [7, 'c', 1],
                    [7, 'c', 1]])                


df = pd.DataFrame(data=my_data, columns=['y', 'dummy', 'x'])
just_dummies = pd.get_dummies(df['dummy'])

step_1 = pd.concat([df, just_dummies], axis=1)      
step_1.drop(['dummy', 'c'], inplace=True, axis=1)
# to run the regression we want to get rid of the strings 'a', 'b', 'c' (obviously)
# and we want to get rid of one dummy variable to avoid the dummy variable trap
# arbitrarily chose "c", coefficients on "a" an "b" would show effect of "a" and "b"
# relative to "c"
step_1 = step_1.applymap(np.int) 

result = sm.OLS(step_1['y'], sm.add_constant(step_1[['x', 'a', 'b']])).fit()
print result.summary()