You can convert most of the columns by just calling convert_objects:
In [36]:
df = df.convert_objects(convert_numeric=True)
df.dtypes
Out[36]:
Date object
WD int64
Manpower float64
2nd object
CTR object
2ndU float64
T1 int64
T2 int64
T3 int64
T4 float64
dtype: object
For column '2nd' and 'CTR' we can call the vectorised str methods to replace the thousands separator and remove the '%' sign and then astype to convert:
In [39]:
df['2nd'] = df['2nd'].str.replace(',','').astype(int)
df['CTR'] = df['CTR'].str.replace('%','').astype(np.float64)
df.dtypes
Out[39]:
Date object
WD int64
Manpower float64
2nd int32
CTR float64
2ndU float64
T1 int64
T2 int64
T3 int64
T4 object
dtype: object
In [40]:
df.head()
Out[40]:
Date WD Manpower 2nd CTR 2ndU T1 T2 T3 T4
0 2013/4/6 6 NaN 2645 5.27 0.29 407 533 454 368
1 2013/4/7 7 NaN 2118 5.89 0.31 257 659 583 369
2 2013/4/13 6 NaN 2470 5.38 0.29 354 531 473 383
3 2013/4/14 7 NaN 2033 6.77 0.37 396 748 681 458
4 2013/4/20 6 NaN 2690 5.38 0.29 361 528 541 381
Or you can do the string handling operations above without the call to astype and then call convert_objects to convert everything in one go.
UPDATE
Since version 0.17.0 convert_objects is deprecated and there isn't a top-level function to do this so you need to do:
df.apply(lambda col:pd.to_numeric(col, errors='coerce'))
See the docs and this related question: pandas: to_numeric for multiple columns
Answer from EdChum on Stack OverflowYou can convert most of the columns by just calling convert_objects:
In [36]:
df = df.convert_objects(convert_numeric=True)
df.dtypes
Out[36]:
Date object
WD int64
Manpower float64
2nd object
CTR object
2ndU float64
T1 int64
T2 int64
T3 int64
T4 float64
dtype: object
For column '2nd' and 'CTR' we can call the vectorised str methods to replace the thousands separator and remove the '%' sign and then astype to convert:
In [39]:
df['2nd'] = df['2nd'].str.replace(',','').astype(int)
df['CTR'] = df['CTR'].str.replace('%','').astype(np.float64)
df.dtypes
Out[39]:
Date object
WD int64
Manpower float64
2nd int32
CTR float64
2ndU float64
T1 int64
T2 int64
T3 int64
T4 object
dtype: object
In [40]:
df.head()
Out[40]:
Date WD Manpower 2nd CTR 2ndU T1 T2 T3 T4
0 2013/4/6 6 NaN 2645 5.27 0.29 407 533 454 368
1 2013/4/7 7 NaN 2118 5.89 0.31 257 659 583 369
2 2013/4/13 6 NaN 2470 5.38 0.29 354 531 473 383
3 2013/4/14 7 NaN 2033 6.77 0.37 396 748 681 458
4 2013/4/20 6 NaN 2690 5.38 0.29 361 528 541 381
Or you can do the string handling operations above without the call to astype and then call convert_objects to convert everything in one go.
UPDATE
Since version 0.17.0 convert_objects is deprecated and there isn't a top-level function to do this so you need to do:
df.apply(lambda col:pd.to_numeric(col, errors='coerce'))
See the docs and this related question: pandas: to_numeric for multiple columns
convert_objects is deprecated.
For pandas >= 0.17.0, use pd.to_numeric
df["2nd"] = pd.to_numeric(df["2nd"])
python - converting a dataframe from float64 to object using dtype - Stack Overflow
python - How to convert float64 time to datetime object - Stack Overflow
grouby( ... , as_index=False) convert "float64" to "object" type.
python - Convert Float to String in Pandas - Stack Overflow
Hi guys, im struggeling with my dataframe.
Im trying to get a percentage of 2 columns, but when I divide the 2 I get this error: unsupported operand type(s) for /: 'float' and 'str'
The dtypes of the columns are an 'object'
Im aware that I have empty data in my columns, but is that an issue?
Would appreciate your help!
If a column contains string or is treated as string, it will have a dtype of object (but not necessarily true backward -- more below). Here is a simple example:
import pandas as pd
df = pd.DataFrame({'SpT': ['string1', 'string2', 'string3'],
'num': ['0.1', '0.2', '0.3'],
'strange': ['0.1', '0.2', 0.3]})
print df.dtypes
#SpT object
#num object
#strange object
#dtype: object
If a column contains only strings, we can apply len on it like what you did should work fine:
print df['num'].apply(lambda x: len(x))
#0 3
#1 3
#2 3
However, a dtype of object does not means it only contains strings. For example, the column strange contains objects with mixed types -- and some str and a float. Applying the function len will raise an error similar to what you have seen:
print df['strange'].apply(lambda x: len(x))
# TypeError: object of type 'float' has no len()
Thus, the problem could be that you have not properly converted the column to string, and the column still contains mixed object types.
Continuing the above example, let us convert strange to strings and check if apply works:
df['strange'] = df['strange'].astype(str)
print df['strange'].apply(lambda x: len(x))
#0 3
#1 3
#2 3
(There is a suspicious discrepancy between df_cleaned and df_clean there in your question, is it a typo or a mistake in the code that causes the problem?)
"Hidden" nulls
If the column dtype is object, TypeError: object of type 'float' has no len() often occurs if the column contains NaN. Check if that's the case by calling
df['Col2'].isna().any()
If it returns True, then there's NaN and you probably need to handle that.
Vectorized str. methods
If null handling is not important, you can also call vectorized str.len(), str.isdigit() etc. methods. For example, the code in the OP can be written as:
df['Col3'] = df['Col2'].str.len().ge(2) & df['Col2'].str[0].str.isalpha()
to get the desired output without errors.
'string' dtype
Since pandas 1.0, there's a new 'string' dtype where you can keep a Nullable integer dtype after casting a column into a 'string' dtype. For example, if you want to convert floats to strings without decimals, yet the column contains NaN values that you want to keep as null, you can use 'string' dtype.
df = pd.DataFrame({
'Col1': [1.2, 3.4, 5.5, float('nan')]
})
df['Col1'] = df['Col1'].astype('string').str.split('.').str[0]
returns
0 1
1 3
2 5
3 <NA>
Name: Col1, dtype: object
where <NA> is a Nullable integer that you can drop with dropna() while df['Col1'].astype(str) casts NaNs into strings.
1. Using pandas.Series.astype
You can use
pandas.Series.astypeYou can do something like this :
weather["Temp"] = weather.Temp.astype(float)
2. Using pd.to_numeric
You can also use
pd.to_numericthat will convert the column from object to floatFor details on how to use it checkout this link :http://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_numeric.html
Example :
s = pd.Series(['apple', '1.0', '2', -3]) print(pd.to_numeric(s, errors='ignore')) print("=========================") print(pd.to_numeric(s, errors='coerce'))Output:
0 apple 1 1.0 2 2 3 -3 ========================= dtype: object 0 NaN 1 1.0 2 2.0 3 -3.0 dtype: float64In your case you can do something like this:
weather["Temp"] = pd.to_numeric(weather.Temp, errors='coerce')
3. Using convert_objects
Other option is to use
convert_objectsExample is as follows
>> pd.Series([1,2,3,4,'.']).convert_objects(convert_numeric=True) 0 1 1 2 2 3 3 4 4 NaN dtype: float64You can use this as follows:
weather["Temp"] = weather.Temp.convert_objects(convert_numeric=True)I have showed you examples because if any of your column won't have a number then it will be converted to
NaN... so be careful while using it.
I tried all methods suggested here but sadly none worked. Instead, found this to be working:
df['column'] = pd.to_numeric(df['column'],errors = 'coerce')
And then check it using:
print(df.info())