You can use to_numeric which will give you a float32
or np.float32(x)
then x.item() will give you python float
Answer from charlie6411 on Stack Overflowpython - sum of 'float64' column type in pandas return float instead of numpy.float64 - Stack Overflow
python - How do i put numpy.float64 data into dataframe? - Stack Overflow
python - Convert a numpy float64 sparse matrix to a pandas data frame - Stack Overflow
BUG: Can't convert `float64[pyarrow]` back to NumPy float
The tolist() method should do what you want. If you have a numpy array, just call tolist():
In [17]: a
Out[17]:
array([ 0. , 0.14285714, 0.28571429, 0.42857143, 0.57142857,
0.71428571, 0.85714286, 1. , 1.14285714, 1.28571429,
1.42857143, 1.57142857, 1.71428571, 1.85714286, 2. ])
In [18]: a.dtype
Out[18]: dtype('float64')
In [19]: b = a.tolist()
In [20]: b
Out[20]:
[0.0,
0.14285714285714285,
0.2857142857142857,
0.42857142857142855,
0.5714285714285714,
0.7142857142857142,
0.8571428571428571,
1.0,
1.1428571428571428,
1.2857142857142856,
1.4285714285714284,
1.5714285714285714,
1.7142857142857142,
1.857142857142857,
2.0]
In [21]: type(b)
Out[21]: list
In [22]: type(b[0])
Out[22]: float
If, in fact, you really have python list of numpy.float64 objects, then @Alexander's answer is great, or you could convert the list to an array and then use the tolist() method. E.g.
In [46]: c
Out[46]:
[0.0,
0.33333333333333331,
0.66666666666666663,
1.0,
1.3333333333333333,
1.6666666666666665,
2.0]
In [47]: type(c)
Out[47]: list
In [48]: type(c[0])
Out[48]: numpy.float64
@Alexander's suggestion, a list comprehension:
In [49]: [float(v) for v in c]
Out[49]:
[0.0,
0.3333333333333333,
0.6666666666666666,
1.0,
1.3333333333333333,
1.6666666666666665,
2.0]
Or, convert to an array and then use the tolist() method.
In [50]: np.array(c).tolist()
Out[50]:
[0.0,
0.3333333333333333,
0.6666666666666666,
1.0,
1.3333333333333333,
1.6666666666666665,
2.0]
If you are concerned with the speed, here's a comparison. The input, x, is a python list of numpy.float64 objects:
In [8]: type(x)
Out[8]: list
In [9]: len(x)
Out[9]: 1000
In [10]: type(x[0])
Out[10]: numpy.float64
Timing for the list comprehension:
In [11]: %timeit list1 = [float(v) for v in x]
10000 loops, best of 3: 109 µs per loop
Timing for conversion to numpy array and then tolist():
In [12]: %timeit list2 = np.array(x).tolist()
10000 loops, best of 3: 70.5 µs per loop
So it is faster to convert the list to an array and then call tolist().
You could use a list comprehension:
floats = [float(np_float) for np_float in np_float_list]
Can you try toarray
pd.DataFrame(A.toarray())
This ipython session shows one way you could do it. The two steps are: convert the sparse matrix to COO format, and then create the Pandas DataFrame using the .row, .col and .data attributes of the COO matrix.
In [50]: data
Out[50]:
<15x15 sparse matrix of type '<class 'numpy.float64'>'
with 11 stored elements in Compressed Sparse Row format>
In [51]: print(data)
(1, 12) 0.8581958095588134
(6, 12) 0.03828052946099181
(6, 14) 0.7908634838351427
(7, 1) 0.7995008873930302
(7, 11) 0.48477191537121145
(7, 13) 0.6226526443518743
(9, 4) 0.37242576669669103
(11, 1) 0.9604278557580955
(11, 5) 0.13285436036287313
(12, 11) 0.5631419223609928
(13, 8) 0.16481624650723847
In [52]: import pandas as pd
In [53]: c = data.tocoo()
In [54]: df = pd.DataFrame({node1: c.row, node2: c.col, edge_weight: c.data})
In [55]: df
Out[55]:
node1 node2 edge_weight
0 1 12 0.858196
1 6 12 0.038281
2 6 14 0.790863
3 7 1 0.799501
4 7 11 0.484772
5 7 13 0.622653
6 9 4 0.372426
7 11 1 0.960428
8 11 5 0.132854
9 12 11 0.563142
10 13 8 0.164816
Use val.item() to convert most NumPy values to a native Python type:
import numpy as np
# for example, numpy.float32 -> python float
val = np.float32(0)
pyval = val.item()
print(type(pyval)) # <class 'float'>
# and similar...
type(np.float64(0).item()) # <class 'float'>
type(np.uint32(0).item()) # <class 'int'>
type(np.int16(0).item()) # <class 'int'>
type(np.cfloat(0).item()) # <class 'complex'>
type(np.datetime64(0, 'D').item()) # <class 'datetime.date'>
type(np.datetime64('2001-01-01 00:00:00').item()) # <class 'datetime.datetime'>
type(np.timedelta64(0, 'D').item()) # <class 'datetime.timedelta'>
...
(A related method np.asscalar(val) was deprecated with 1.16, and removed with 1.23).
For the curious, to build a table of conversions of NumPy array scalars for your system:
for name in dir(np):
obj = getattr(np, name)
if hasattr(obj, 'dtype'):
try:
if 'time' in name:
npn = obj(0, 'D')
else:
npn = obj(0)
nat = npn.item()
print('{0} ({1!r}) -> {2}'.format(name, npn.dtype.char, type(nat)))
except:
pass
There are a few NumPy types that have no native Python equivalent on some systems, including: clongdouble, clongfloat, complex192, complex256, float128, longcomplex, longdouble and longfloat. These need to be converted to their nearest NumPy equivalent before using .item().
If you want to convert (numpy.array OR numpy scalar OR native type OR numpy.darray) TO native type you can simply do :
converted_value = getattr(value, "tolist", lambda: value)()
tolist will convert your scalar or array to python native type. The default lambda function takes care of the case where value is already native.
Solution for pandas 0.24+ for converting numeric with missing values:
df = pd.DataFrame({'column name':[7500000.0,7500000.0, np.nan]})
print (df['column name'])
0 7500000.0
1 7500000.0
2 NaN
Name: column name, dtype: float64
df['column name'] = df['column name'].astype(np.int64)
ValueError: Cannot convert non-finite values (NA or inf) to integer
#http://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html
df['column name'] = df['column name'].astype('Int64')
print (df['column name'])
0 7500000
1 7500000
2 NaN
Name: column name, dtype: Int64
I think you need cast to numpy.int64:
df['column name'].astype(np.int64)
Sample:
df = pd.DataFrame({'column name':[7500000.0,7500000.0]})
print (df['column name'])
0 7500000.0
1 7500000.0
Name: column name, dtype: float64
df['column name'] = df['column name'].astype(np.int64)
#same as
#df['column name'] = df['column name'].astype(pd.np.int64)
print (df['column name'])
0 7500000
1 7500000
Name: column name, dtype: int64
If some NaNs in columns need replace them to some int (e.g. 0) by fillna, because type of NaN is float:
df = pd.DataFrame({'column name':[7500000.0,np.nan]})
df['column name'] = df['column name'].fillna(0).astype(np.int64)
print (df['column name'])
0 7500000
1 0
Name: column name, dtype: int64
Also check documentation - missing data casting rules
EDIT:
Convert values with NaNs is buggy:
df = pd.DataFrame({'column name':[7500000.0,np.nan]})
df['column name'] = df['column name'].values.astype(np.int64)
print (df['column name'])
0 7500000
1 -9223372036854775808
Name: column name, dtype: int64
You can need to pass in the string 'int64':
>>> import pandas as pd
>>> df = pd.DataFrame({'a': [1.0, 2.0]}) # some test dataframe
>>> df['a'].astype('int64')
0 1
1 2
Name: a, dtype: int64
There are some alternative ways to specify 64-bit integers:
>>> df['a'].astype('i8') # integer with 8 bytes (64 bit)
0 1
1 2
Name: a, dtype: int64
>>> import numpy as np
>>> df['a'].astype(np.int64) # native numpy 64 bit integer
0 1
1 2
Name: a, dtype: int64
Or use np.int64 directly on your column (but it returns a numpy.array):
>>> np.int64(df['a'])
array([1, 2], dtype=int64)
Yes, actually when you use Python's native float to specify the dtype for an array , numpy converts it to float64. As given in documentation -
Note that, above, we use the Python float object as a dtype. NumPy knows that
intrefers tonp.int_,boolmeansnp.bool_, thatfloatisnp.float_andcomplexisnp.complex_. The other data-types do not have Python equivalents.
And -
float_ - Shorthand for float64.
This is why even though you use float to convert the whole array to float , it still uses np.float64.
According to the requirement from the other question , the best solution would be converting to normal float object after taking each scalar value as -
float(new_array[0])
A solution that I could think of is to create a subclass for float and use that for casting (though to me it looks bad). But I would prefer the previous solution over this if possible. Example -
In [20]: import numpy as np
In [21]: na = np.array([1., 2., 3.])
In [22]: na = np.array([1., 2., 3., np.inf, np.inf])
In [23]: type(na[-1])
Out[23]: numpy.float64
In [24]: na[-1] - na[-2]
C:\Anaconda3\Scripts\ipython-script.py:1: RuntimeWarning: invalid value encountered in double_scalars
if __name__ == '__main__':
Out[24]: nan
In [25]: class x(float):
....: pass
....:
In [26]: na_new = na.astype(x)
In [28]: type(na_new[-1])
Out[28]: float #No idea why its showing float, I would have thought it would show '__main__.x' .
In [29]: na_new[-1] - na_new[-2]
Out[29]: nan
In [30]: na_new
Out[30]: array([1.0, 2.0, 3.0, inf, inf], dtype=object)
You can create an anonymous type float like this
>>> new_array = my_array.astype(type('float', (float,), {}))
>>> type(new_array[0])
<type 'float'>
Hi,
In the example below I want to change the value in column B based on the value in column A.
Why doesn't the value change in B?
import pandas as pd
import numpy as np
data = {
"A": [15, np.nan, None, 4],
"B": [15, 5, 7, 4],
}
df = pd.DataFrame(data)
df = df.astype({"A": "float64", "B": "float64"})
print(df)
print(df.dtypes)
# none of these work. why?
df.loc[df['A'] == None, 'B'] = np.nan # if col A is None then set col B to np.nan
df.loc[df['A'] == np.nan, 'B'] = np.nan # if col A is np.nan then set col B to np.nan
df.loc[df['A'] == pd.NA, 'B'] = np.nan # if col A is pd.NA then set col B to np.nan
df.loc[df['A'] == float("nan"), 'B'] = np.nan # if col A is float("nan") then set col B to np.nan
print(df)Thanks
If you don't see the point (no data loss) you can manually downcast the column to a standard numpy type by passing the column values through a numpy array and changing its type, here: to numpy.float64 (which reconstructs also the index):
df[col_name] = df[col_name].values.astype(float)
In [52]: pd.Float64Dtype?
Init signature: pd.Float64Dtype()
Docstring:
An ExtensionDtype for float64 data.
This dtype uses ``pd.NA`` as missing value indicator.
With a float dtype, the frame displays as
In [68]: df
Out[68]:
val1 val2 val3
0 1.0 2.0 3.0
1 4.0 5.0 6.0
2 7.0 8.0 9.0
3 10.0 NaN 12.0
4 13.0 14.0 15.0
where the NaN is the np.nan, a valid float.
In [71]: df
Out[71]:
val1 val2 val3
0 1.0 2.0 3.0
1 4.0 5.0 6.0
2 7.0 8.0 9.0
3 10.0 <NA> 12.0
4 13.0 14.0 15.0
where that <NA> is pandas._libs.missing.NAType
Your df.interpolate() error indicates that the extension dtype was not implemented for all operations. Some places suggest it is still experimental.