Documenting the answer that worked for me based on the comment by @piRSquared.
I needed to convert to a string first, then an integer.
>>> df['purchase'].astype(str).astype(int)
Documenting the answer that worked for me based on the comment by @piRSquared.
I needed to convert to a string first, then an integer.
>>> df['purchase'].astype(str).astype(int)
pandas >= 1.0
convert_dtypes
The (self) accepted answer doesn't take into consideration the possibility of NaNs in object columns.
df = pd.DataFrame({
'a': [1, 2, np.nan],
'b': [True, False, np.nan]}, dtype=object)
df
a b
0 1 True
1 2 False
2 NaN NaN
df['a'].astype(str).astype(int) # raises ValueError
This chokes because the NaN is converted to a string "nan", and further attempts to coerce to integer will fail. To avoid this issue, we can soft-convert columns to their corresponding nullable type using convert_dtypes:
df.convert_dtypes()
a b
0 1 True
1 2 False
2 <NA> <NA>
df.convert_dtypes().dtypes
a Int64
b boolean
dtype: object
If your data has junk text mixed in with your ints, you can use pd.to_numeric as an initial step:
s = pd.Series(['1', '2', '...'])
s.convert_dtypes() # converts to string, which is not what we want
0 1
1 2
2 ...
dtype: string
# coerces non-numeric junk to NaNs
pd.to_numeric(s, errors='coerce')
0 1.0
1 2.0
2 NaN
dtype: float64
# one final `convert_dtypes` call to convert to nullable int
pd.to_numeric(s, errors='coerce').convert_dtypes()
0 1
1 2
2 <NA>
dtype: Int64
How do you change object to int in dataset
Coverting data type of a df column from obj to int
Reading through the documents for sklearn. This package has thorough documentation. In particular the Preprocessing section on encoding categorical features:
In regards to keeping categorical features represented in an array of integers, ie [1,2,3,4,5], we have this:
Such integer representation can not be used directly with scikit-learn estimators, as these expect continuous input, and would interpret the categories as being ordered, which is often not desired (i.e. the set of browsers was ordered arbitrarily). One possibility to convert categorical features to features that can be used with scikit-learn estimators is to use a one-of-K or one-hot encoding, which is implemented in OneHotEncoder. This estimator transforms each categorical feature with m possible values into m binary features, with only one active.
So what you can to do is convert your array into 5 new columns (this case, since you have 5 possible values) using one-hot encoding.
Here is some working code. The input is a column of categorical parameters [1,2,3,4,5], the ouput is a matrix, 5 columns, 1 for each of the 5 possible choices:
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder()
enc.fit([[1],[2],[3],[4],[5]])
OneHotEncoder(categorical_features='all', dtype='numpy.float64', handle_unknown='error', n_values='auto', sparse=True)
print enc.transform([[1],[2],[3],[4],[5]]).toarray()
Output:
[[ 1. 0. 0. 0. 0.]
[ 0. 1. 0. 0. 0.]
[ 0. 0. 1. 0. 0.]
[ 0. 0. 0. 1. 0.]
[ 0. 0. 0. 0. 1.]]
Say your categorical parameters were in this order: [1,3,2,5,4,3,2,1,3,4,2]. You would get this output:
[[ 1. 0. 0. 0. 0.]
[ 0. 0. 1. 0. 0.]
[ 0. 1. 0. 0. 0.]
[ 0. 0. 0. 0. 1.]
[ 0. 0. 0. 1. 0.]
[ 0. 0. 1. 0. 0.]
[ 0. 1. 0. 0. 0.]
[ 1. 0. 0. 0. 0.]
[ 0. 0. 1. 0. 0.]
[ 0. 0. 0. 1. 0.]
[ 0. 1. 0. 0. 0.]]
So this 1 column will convert into 5 columns.
print(data.dtypes)
data["col_name"]=data["col_name"].astype(str)
print(data.dtypes)
In the dataset that I am trying to process with python include Dtypes: object/int64/float64 and I am trying to change all the data to int64. The problem I think is that one of my data column include both numbers and letters (W19S01) so I get the following error when I try to convert the Dtype: ValueError: invalid literal for int() with base 10: 'W19S01'
So far I have tried to change the data type to string using df[cols] = df[cols].to_string().strip() then changing it back to float/int64 and it still shows ValueError: could not convert string to float: 'W19S01'
I havealso tried to replace the letters in data set with empty space using df['Subject Yr Student ID'].str.replace('W', '').str.replace('S', '')
So far no luck and I am fairly new to python