You can use this solution implemented to pandas by Series.apply:
from Crypto.Cipher import XOR
import base64
def encrypt(key, plaintext):
cipher = XOR.new(key)
return base64.b64encode(cipher.encrypt(plaintext))
def decrypt(key, ciphertext):
cipher = XOR.new(key)
return cipher.decrypt(base64.b64decode(ciphertext))
load['Encoded_Column'] = load['F'].apply(lambda x: encrypt('password',x))
load['Decoded_Column'] = (load['Encoded_Column'].apply(lambda x: decrypt('password', x))
.str.decode("utf-8"))
print (load)
A B C D E F Encoded_Column Decoded_Column
0 a 4 7 1 5 a b'EQ==' a
1 b 5 8 3 3 a b'EQ==' a
2 c 4 9 5 6 a b'EQ==' a
3 d 5 4 4 9 b b'Eg==' b
4 e 5 2 2 2 b b'Eg==' b
5 f 4 0 0 4 b b'Eg==' b
Another solution:
import base64
def encode(key, clear):
enc = []
for i in range(len(clear)):
key_c = key[i % len(key)]
enc_c = chr((ord(clear[i]) + ord(key_c)) % 256)
enc.append(enc_c)
return base64.urlsafe_b64encode("".join(enc).encode()).decode()
def decode(key, enc):
dec = []
enc = base64.urlsafe_b64decode(enc).decode()
for i in range(len(enc)):
key_c = key[i % len(key)]
dec_c = chr((256 + ord(enc[i]) - ord(key_c)) % 256)
dec.append(dec_c)
return "".join(dec)
load['Encoded_Column'] = load['F'].apply(lambda x: encode('password',x))
load['Decoded_Column'] = load['Encoded_Column'].apply(lambda x: decode('password', x))
Or use list comprehension:
load['Encoded_Column'] = [encode('password',x) for x in load['F']]
load['Decoded_Column'] = [decode('password', x) for x in load['Encoded_Column']]
print (load)
A B C D E F Encoded_Column Decoded_Column
0 a 4 7 1 5 a w5E= a
1 b 5 8 3 3 a w5E= a
2 c 4 9 5 6 a w5E= a
3 d 5 4 4 9 b w5I= b
4 e 5 2 2 2 b w5I= b
5 f 4 0 0 4 b w5I= b
Answer from jezrael on Stack OverflowYou can use this solution implemented to pandas by Series.apply:
from Crypto.Cipher import XOR
import base64
def encrypt(key, plaintext):
cipher = XOR.new(key)
return base64.b64encode(cipher.encrypt(plaintext))
def decrypt(key, ciphertext):
cipher = XOR.new(key)
return cipher.decrypt(base64.b64decode(ciphertext))
load['Encoded_Column'] = load['F'].apply(lambda x: encrypt('password',x))
load['Decoded_Column'] = (load['Encoded_Column'].apply(lambda x: decrypt('password', x))
.str.decode("utf-8"))
print (load)
A B C D E F Encoded_Column Decoded_Column
0 a 4 7 1 5 a b'EQ==' a
1 b 5 8 3 3 a b'EQ==' a
2 c 4 9 5 6 a b'EQ==' a
3 d 5 4 4 9 b b'Eg==' b
4 e 5 2 2 2 b b'Eg==' b
5 f 4 0 0 4 b b'Eg==' b
Another solution:
import base64
def encode(key, clear):
enc = []
for i in range(len(clear)):
key_c = key[i % len(key)]
enc_c = chr((ord(clear[i]) + ord(key_c)) % 256)
enc.append(enc_c)
return base64.urlsafe_b64encode("".join(enc).encode()).decode()
def decode(key, enc):
dec = []
enc = base64.urlsafe_b64decode(enc).decode()
for i in range(len(enc)):
key_c = key[i % len(key)]
dec_c = chr((256 + ord(enc[i]) - ord(key_c)) % 256)
dec.append(dec_c)
return "".join(dec)
load['Encoded_Column'] = load['F'].apply(lambda x: encode('password',x))
load['Decoded_Column'] = load['Encoded_Column'].apply(lambda x: decode('password', x))
Or use list comprehension:
load['Encoded_Column'] = [encode('password',x) for x in load['F']]
load['Decoded_Column'] = [decode('password', x) for x in load['Encoded_Column']]
print (load)
A B C D E F Encoded_Column Decoded_Column
0 a 4 7 1 5 a w5E= a
1 b 5 8 3 3 a w5E= a
2 c 4 9 5 6 a w5E= a
3 d 5 4 4 9 b w5I= b
4 e 5 2 2 2 b w5I= b
5 f 4 0 0 4 b w5I= b
import pandas as pd
import binascii
load = pd.DataFrame({'A':list('abcdef'),
'B':[4,5,4,5,5,4],
'C':[7,8,9,4,2,0],
'D':[1,3,5,4,2,0],
'E':[5,3,6,9,2,4],
'F':[binascii.hexlify(x.encode()) for x in 'aaabbb']
})
A B C D E F
0 a 4 7 1 5 b'61'
1 b 5 8 3 3 b'61'
2 c 4 9 5 6 b'61'
3 d 5 4 4 9 b'62'
4 e 5 2 2 2 b'62'
5 f 4 0 0 4 b'62'
# decode
binascii.unhexlify(load.loc[1]['F']).decode('utf-8') -->> 'a'
example
print(binascii.hexlify('HelloWorld'.encode())) --> b'48656c6c6f576f726c64'
print(binascii.unhexlify('48656c6c6f576f726c64'.encode())) --> b'HelloWorld'
You can using category dtype in sklearn , it should be labelencoder
df.city=df.city.astype('category').cat.codes
df
Out[385]:
school city category capacity
0 1 0 45 23
1 2 1 12 236
2 3 2 8 63
3 4 0 7 234
A few thousand columns is still manageable in the context of ML classifiers. Although you'd want to watch out for the curse of dimensionality.
That aside, you wouldn't want a get_dummies call to result in a memory blowout, so you could generate a SparseDataFrame instead -
v = pd.get_dummies(df.set_index('school').city, sparse=True)
v
azez6576sebd dsqozbc765aj sqdqsd12887s
school
1 1 0 0
2 0 1 0
3 0 0 1
4 1 0 0
type(v)
pandas.core.sparse.frame.SparseDataFrame
You can generate a sparse matrix using sdf.to_coo -
v.to_coo()
<4x3 sparse matrix of type '<class 'numpy.uint8'>'
with 4 stored elements in COOrdinate format>
Encoding
As the first error says, codecs is not callable. In fact is the name of the module.
You probably want:
data['text'] = data.apply(lambda row:
codecs.encode(row['text'], 'utf-8'), axis=1)
Tokenization
The error raised by word_tokenize is due to the fact that the function is used on the previously encoded string: codecs.encode renders the text into a bytes literal string.
From the codecs doc:
Most standard codecs are text encodings, which encode text to bytes, but there are also codecs provided that encode text to text, and bytes to bytes.
word_tokenize doesn't work with bytes literar, like the error says (last line of your error traceback).
If you remove the encoding passage it will work.
About your worries on the video: the prefix u means unicode.1
The prefix b means bytes literal.2 This is the prefix of the strings if you print your dataframe after the use of codecs.encode.
In python 3 (I see from the traceback that your version is 3.6) the default string type is Unicode, so the u is redundant and often not shown, but the strings are already unicode.
So I'm quite sure you are safe: you can safely not use codecs.encode.
You could even do something simpler:
df['text'] = df['text'].str.encode('utf-8')
Ref: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.encode.html
If you're not using csv, and you want to encode your string index, this is what worked for me:
df.index = df.index.str.encode('utf-8')
Setting up the encoding should be treated when reading the input file, using the option encoding
df = pd.read_csv('bibliography.csv', delimiter=',', encoding="utf-8")
or if the file uses BOM,
df = pd.read_csv('bibliography.csv', delimiter=',', encoding="utf-8-sig")
One way would be to use df.replace(). You would avoid changing numeric columns this way.
df.replace('[A-Za-z]','N', regex=True).replace('\d','D', regex=True)
Full example with a numeric column called D, A non-numeric called N and TransDetails.
import pandas as pd
data = '''\
D,N,TransDetails
1,ABC,NEFT-PUNB0315500-JITENDER SING
1,123,NEFT-UTIB0CCH274-VIRENDER KUMA
1,123,NEFT-UTIB0CCH274-SUNITA DEVI
1,123,NEFT-PUNB0315500-AMLASH KUMAR
1,123,NEFT-PUNB0109800-FARIDUDDEN
1,123,NEFT-PUNB0109800-IDREESH
1,123,NEFT-PUNB0315500-BUDDHU
1,123,NEFT-UTIB0CCH274-SAKIL AHAMAD
1,123,NEFT-UTIB0CCH274-NAIM AHAMAD
1,123,NEFT-UTIB0CCH274-SALIM AHAMAD
1,123,NEFT-UTIB0CCH274-NADIM AHAMAD'''
fileobj = pd.compat.StringIO(data) # or 'path/to/csv'
df = pd.read_csv(fileobj)
df = df.replace('[A-Za-z]','N', regex=True).replace('\d','D', regex=True)
print(df)
Returns:
D N TransDetails
0 1 NNN NNNN-NNNNDDDDDDD-NNNNNNNN NNNN
1 1 DDD NNNN-NNNNDNNNDDD-NNNNNNNN NNNN
2 1 DDD NNNN-NNNNDNNNDDD-NNNNNN NNNN
3 1 DDD NNNN-NNNNDDDDDDD-NNNNNN NNNNN
4 1 DDD NNNN-NNNNDDDDDDD-NNNNNNNNNN
5 1 DDD NNNN-NNNNDDDDDDD-NNNNNNN
6 1 DDD NNNN-NNNNDDDDDDD-NNNNNN
7 1 DDD NNNN-NNNNDNNNDDD-NNNNN NNNNNN
8 1 DDD NNNN-NNNNDNNNDDD-NNNN NNNNNN
9 1 DDD NNNN-NNNNDNNNDDD-NNNNN NNNNNN
10 1 DDD NNNN-NNNNDNNNDDD-NNNNN NNNNNN
You can use a regular expression to handle the replacement.
df['TransDetails'] = df['TransDetails'].str.replace('[A-Za-z]', 'N')
df['TransDetails'] = df['TransDetails'].str.replace('\d', 'D')
df
# returns:
TransDetails
0 NNNN-NNNNDDDDDDD-NNNNNNNN NNNN
1 NNNN-NNNNDNNNDDD-NNNNNNNN NNNN
2 NNNN-NNNNDNNNDDD-NNNNNN NNNN
3 NNNN-NNNNDDDDDDD-NNNNNN NNNNN
4 NNNN-NNNNDDDDDDD-NNNNNNNNNN
5 NNNN-NNNNDDDDDDD-NNNNNNN
6 NNNN-NNNNDDDDDDD-NNNNNN
7 NNNN-NNNNDNNNDDD-NNNNN NNNNNN
8 NNNN-NNNNDNNNDDD-NNNN NNNNNN
9 NNNN-NNNNDNNNDDD-NNNNN NNNNNN
10 NNNN-NNNNDNNNDDD-NNNNN NNNNNN
You can easily do this though,
df.apply(LabelEncoder().fit_transform)
EDIT2:
In scikit-learn 0.20, the recommended way is
OneHotEncoder().fit_transform(df)
as the OneHotEncoder now supports string input. Applying OneHotEncoder only to certain columns is possible with the ColumnTransformer.
EDIT:
Since this original answer is over a year ago, and generated many upvotes (including a bounty), I should probably extend this further.
For inverse_transform and transform, you have to do a little bit of hack.
from collections import defaultdict
d = defaultdict(LabelEncoder)
With this, you now retain all columns LabelEncoder as dictionary.
# Encoding the variable
fit = df.apply(lambda x: d[x.name].fit_transform(x))
# Inverse the encoded
fit.apply(lambda x: d[x.name].inverse_transform(x))
# Using the dictionary to label future data
df.apply(lambda x: d[x.name].transform(x))
MOAR EDIT:
Using Neuraxle's FlattenForEach step, it's possible to do this as well to use the same LabelEncoder on all the flattened data at once:
FlattenForEach(LabelEncoder(), then_unflatten=True).fit_transform(df)
For using separate LabelEncoders depending for your columns of data, or if only some of your columns of data needs to be label-encoded and not others, then using a ColumnTransformer is a solution that allows for more control on your column selection and your LabelEncoder instances.
As mentioned by larsmans, LabelEncoder() only takes a 1-d array as an argument. That said, it is quite easy to roll your own label encoder that operates on multiple columns of your choosing, and returns a transformed dataframe. My code here is based in part on Zac Stewart's excellent blog post found here.
Creating a custom encoder involves simply creating a class that responds to the fit(), transform(), and fit_transform() methods. In your case, a good start might be something like this:
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.pipeline import Pipeline
# Create some toy data in a Pandas dataframe
fruit_data = pd.DataFrame({
'fruit': ['apple','orange','pear','orange'],
'color': ['red','orange','green','green'],
'weight': [5,6,3,4]
})
class MultiColumnLabelEncoder:
def __init__(self,columns = None):
self.columns = columns # array of column names to encode
def fit(self,X,y=None):
return self # not relevant here
def transform(self,X):
'''
Transforms columns of X specified in self.columns using
LabelEncoder(). If no columns specified, transforms all
columns in X.
'''
output = X.copy()
if self.columns is not None:
for col in self.columns:
output[col] = LabelEncoder().fit_transform(output[col])
else:
for colname,col in output.iteritems():
output[colname] = LabelEncoder().fit_transform(col)
return output
def fit_transform(self,X,y=None):
return self.fit(X,y).transform(X)
Suppose we want to encode our two categorical attributes (fruit and color), while leaving the numeric attribute weight alone. We could do this as follows:
MultiColumnLabelEncoder(columns = ['fruit','color']).fit_transform(fruit_data)
Which transforms our fruit_data dataset from
to

Passing it a dataframe consisting entirely of categorical variables and omitting the columns parameter will result in every column being encoded (which I believe is what you were originally looking for):
MultiColumnLabelEncoder().fit_transform(fruit_data.drop('weight',axis=1))
This transforms
to
.
Note that it'll probably choke when it tries to encode attributes that are already numeric (add some code to handle this if you like).
Another nice feature about this is that we can use this custom transformer in a pipeline:
encoding_pipeline = Pipeline([
('encoding',MultiColumnLabelEncoder(columns=['fruit','color']))
# add more pipeline steps as needed
])
encoding_pipeline.fit_transform(fruit_data)