replace 'na' with 'nan' so it is then converted to np.nan, then use np.nanmean.
example:
test = np.array(['0','1','nan'], dtype=float)
np.where(np.isnan(test), np.nanmean(test), test)
array([ 0. , 1. , 0.5])
Answer from Julien on Stack Overflowreplace 'na' with 'nan' so it is then converted to np.nan, then use np.nanmean.
example:
test = np.array(['0','1','nan'], dtype=float)
np.where(np.isnan(test), np.nanmean(test), test)
array([ 0. , 1. , 0.5])
Better would be to first convert 'na' to proper NaN. Then one can use the data anyway one wants:
import numpy as np
val_inc[val_inc == 'na'] = np.nan # 'na' to proper NaN or missing value
val_inc = val_inc.astype(np.float) # no error here now.
print(val_inc)
Ouput:
[ nan 38.012 38.7816 38.0736 40.7118 44.7382 39.6416 38.9177
36.9031 43.2611 38.2732 40.7129 37.2844 39.5835 43.9194 42.5485
36.9052 nan 41.9264 45.3568 44.6239 38.1079 45.2393 32.785
44.6239 38.0216 38.4608 42.5644 35.3127 33.2936 33.0556 40.4476
35.6581 35.5574 43.1096 34.4751 42.0554 40.3944 40.2466 32.2567
nan 38.8594 43.947 41.7973 41.8105 40.3797 31.2868 45.3644
40.7177 41.8558 38.9249 33.2077 42.4053 42.559 ]
Use numpy.concatenate:
>>> arr = array([array([[2.4567]],dtype=object),array([[3.4567]],dtype=object),array([[4.4567]],dtype=object),array([[5.4567]],dtype=object),array([[6.4567]], dtype=object)])
>>> np.concatenate(arr).astype(None)
array([[ 2.4567],
[ 3.4567],
[ 4.4567],
[ 5.4567],
[ 6.4567]])
You can use np.stack, works also for the multi dimensional case:
import numpy as np
from numpy import array
arr = array([array([[2.4567]],dtype=object),array([[3.4567]],dtype=object),array([[4.4567]],dtype=object),array([[5.4567]],dtype=object),array([[6.4567]],
dtype=object)])
np.stack(arr).astype(None)
array([[[2.4567]],
[[3.4567]],
[[4.4567]],
[[5.4567]],
[[6.4567]]])
If you have a list of lists, you only needed to use ...
import numpy as np
...
npa = np.asarray(someListOfLists, dtype=np.float32)
per this LINK in the scipy / numpy documentation. You just needed to define dtype inside the call to asarray.
If you're converting a list into an array, you'll need to make a new copy anyway, so
arr = np.array(my_list, dtype='float32')
also works.
One use case is when my_list is not actually a list but some other list-like object; in which case explicitly casting to list beforehand might be helpful.
arr = np.array(list(my_list), dtype='float32')
For example,
my_list = pd.Series([[1], [2], [3]]).values
np.array(my_list) # jagged array; not OK
np.array(list(my_list)) # OK
np.array(my_list.tolist()) # OK
my_list = {'a': [1], 'b': [2]}.values()
np.array(my_list) # jagged array; not OK
np.array(list(my_list)) # OK
To cast nested list into an array, the shapes of the sublists must match. If they don't maybe you want to concatenate the sublists along some axis. Try np.concatenate/np.r_/np.c_ etc. instead.
Nasty little problem... I have been fooling around with this toy example:
>>> arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
>>> arr
array([['one', [1, 2, 3]],
['two', [4, 5, 6]]], dtype=object)
My first guess was:
>>> np.array(arr[:, 1])
array([[1, 2, 3], [4, 5, 6]], dtype=object)
But that keeps the object dtype, so perhaps then:
>>> np.array(arr[:, 1], dtype=float)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.
You can normally work around this doing the following:
>>> np.array(arr[:, 1], dtype=[('', float)]*3).view(float).reshape(-1, 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected a readable buffer object
Not here though, which was kind of puzzling. Apparently it is the fact that the objects in your array are lists that throws this off, as replacing the lists with tuples works:
>>> np.array([tuple(j) for j in arr[:, 1]],
... dtype=[('', float)]*3).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Since there doesn't seem to be any entirely satisfactory solution, the easiest is probably to go with:
>>> np.array(list(arr[:, 1]), dtype=float)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Although that will not be very efficient, probably better to go with something like:
>>> np.fromiter((tuple(j) for j in arr[:, 1]), dtype=[('', float)]*3,
... count=len(arr)).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Based on Jaime's toy example I think you can do this very simply using np.vstack():
arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
float_arr = np.vstack(arr[:, 1]).astype(np.float)
This will work regardless of whether the 'numeric' elements in your object array are 1D numpy arrays, lists or tuples.