Nasty little problem... I have been fooling around with this toy example:
>>> arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
>>> arr
array([['one', [1, 2, 3]],
['two', [4, 5, 6]]], dtype=object)
My first guess was:
>>> np.array(arr[:, 1])
array([[1, 2, 3], [4, 5, 6]], dtype=object)
But that keeps the object dtype, so perhaps then:
>>> np.array(arr[:, 1], dtype=float)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.
You can normally work around this doing the following:
>>> np.array(arr[:, 1], dtype=[('', float)]*3).view(float).reshape(-1, 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected a readable buffer object
Not here though, which was kind of puzzling. Apparently it is the fact that the objects in your array are lists that throws this off, as replacing the lists with tuples works:
>>> np.array([tuple(j) for j in arr[:, 1]],
... dtype=[('', float)]*3).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Since there doesn't seem to be any entirely satisfactory solution, the easiest is probably to go with:
>>> np.array(list(arr[:, 1]), dtype=float)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Although that will not be very efficient, probably better to go with something like:
>>> np.fromiter((tuple(j) for j in arr[:, 1]), dtype=[('', float)]*3,
... count=len(arr)).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Nasty little problem... I have been fooling around with this toy example:
>>> arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
>>> arr
array([['one', [1, 2, 3]],
['two', [4, 5, 6]]], dtype=object)
My first guess was:
>>> np.array(arr[:, 1])
array([[1, 2, 3], [4, 5, 6]], dtype=object)
But that keeps the object dtype, so perhaps then:
>>> np.array(arr[:, 1], dtype=float)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.
You can normally work around this doing the following:
>>> np.array(arr[:, 1], dtype=[('', float)]*3).view(float).reshape(-1, 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected a readable buffer object
Not here though, which was kind of puzzling. Apparently it is the fact that the objects in your array are lists that throws this off, as replacing the lists with tuples works:
>>> np.array([tuple(j) for j in arr[:, 1]],
... dtype=[('', float)]*3).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Since there doesn't seem to be any entirely satisfactory solution, the easiest is probably to go with:
>>> np.array(list(arr[:, 1]), dtype=float)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Although that will not be very efficient, probably better to go with something like:
>>> np.fromiter((tuple(j) for j in arr[:, 1]), dtype=[('', float)]*3,
... count=len(arr)).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Based on Jaime's toy example I think you can do this very simply using np.vstack():
arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
float_arr = np.vstack(arr[:, 1]).astype(np.float)
This will work regardless of whether the 'numeric' elements in your object array are 1D numpy arrays, lists or tuples.
Yet another way
np.frompyfunc(lambda x: x.replace(',',''),1,1)(arr).astype(float)
frompyfunc returns an object dtype array, which is fine in this case. Often I've found that it is 2x faster than than a list comprehension, but here it times about the same as @coldspeed's:
np.array([v.replace(',', '') for v in arr], dtype=np.float32)
That may be because we are starting with an object dtype array. Direct iteration on an object dtype is a bit slower than iteration on a list, but faster than iteration on a regular numpy array. Like a list, the elements of the array are pointers to strings, and don't require the 'unboxing' that a string dtype array would.
(and 2 to 3 x faster than the np.char version).
Simple way to do it is remove every comma:
np.array([v.replace(',', '') for v in arr], dtype=np.float32)
If you have pandas, to_numeric is a good option. It gracefully handles any invalid values that may creep in post replacement.
pd.to_numeric([v.replace(',', '') for v in arr], errors='coerce', downcast='float')
Both methods return a float array as output.
Short answer:
M_rref.astype(float) will fix your problem.
Long answer:
Why does this happen?
In order to find out, try going back to your matrix M and try the following:
>>> print(M[0])
>>> print(type(M[0]))
You will get:
4
sympy.core.numbers.Integer
That's why. Even though it says it's 4, it's actually a SymPy object that is stored inside the SymPy matrices. And that's the way it should be: SymPy is for symbolic mathematics. Therefore, when you convert it to a numpy array, NumPy will recognize sympy.core.numbers.Integer as an object.
Hope that answers your question.
In this PR there is a note about incorrect conversion of numpy matrices to SymPy matrices. Using that (and not float, which I don' think is intended to convert contents of an object to floats) I would do:
>>> sy.Matrix(M.rows, M.cols, [sympify(x) for x in M_rref]).n()
This should give you a SymPy matrix with entries evaluated as floats.
131.199997,1.31199997e+02 are equivalent displays of the same number. They are both 'normal floats'.
In:
array([[131.199997, 137.149994, 131.199997, 136.320007, 117.08859299999999,
615015],
[135.199997, 136.570007, 134.330002, 135.639999, 116.504501, 352835],
[131.419998, 133.5, 130.759995, 131.779999, 113.18906399999999,
209805]], dtype=object)
each element is a Python float, and is formatted individually, regardless of the value. Note how some strings are long, other much short.
In:
a = a.astype('float64') # converting to `float64`
a
array([[ 1.31199997e+02, 1.37149994e+02, 1.31199997e+02,
1.36320007e+02, 1.17088593e+02, 6.15015000e+05],
[ 1.35199997e+02, 1.36570007e+02, 1.34330002e+02,
1.35639999e+02, 1.16504501e+02, 3.52835000e+05],
[ 1.31419998e+02, 1.33500000e+02, 1.30759995e+02,
1.31779999e+02, 1.13189064e+02, 2.09805000e+05]])
the array is displayed as a whole, using a format that will work equally well for the smaller values (1e2, 100) and the larger ones (1e5, 100000). With this format it use neat columns, show the 2d array structure.
While you can control how numpy displays such an array, it does not change the underlying numeric values. And for fast numpy calculations, you want this numeric dtype, not the object one.
Try df.sample(3).values[:,1:-1]. That should just be the float values that are all around 100. It's the last integer column with values like 209805 that's triggering the scientific notation.
Better yet, select the columns from the dataframe, "Open,High,Low,Close,Adj Close" before applying .values. Those are all float64 dtype, and the resulting array will have that dtype as well. Select the integer volume column separately. You are already treating the string/object date column separately.
Try:
a = df[df.columns[1:-1]].sample().values
Try adding:
np.set_printoptions(suppress=True)
As the first line under import numpy as np.
Lets take as the starting point a list of lists or equivalently an object array of lists:
A = [[], [], [], [1,2,1], [], [2, 2, 0, 0], []]
A = array([[], [], [], [1,2,1], [], [2, 2, 0, 0], []], dtype=object)
If the sublists were all the same length, np.array([...]) would give you a 2d array, one row for each sublist, and columns matching their common length. But since they are unequal in length, it can only make it a 1d array, where each element is a pointer to one of these sublists - i.e. dtype=object.
I can imagine 2 ways of constructing a 2d array:
- pad each sublist to a common length
- insert each sublist into an empty array of the appropriate size.
Basically it requires common Python iteration; it's not a common enough task to have a wiz-bang numpy function.
For example:
In [346]: n=len(A)
In [348]: m=max([len(x) for x in A])
In [349]: AA=np.zeros((n,m),int)
In [350]: for i,x in enumerate(A):
.....: AA[i,:len(x)] = x
In [351]: AA
Out[351]:
array([[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[1, 2, 1, 0],
[0, 0, 0, 0],
[2, 2, 0, 0],
[0, 0, 0, 0]])
To get a sparse matrix:
In [352]: from scipy import sparse
In [353]: MA=sparse.coo_matrix(AA)
In [354]: MA
Out[354]:
<7x4 sparse matrix of type '<class 'numpy.int32'>'
with 5 stored elements in COOrdinate format>
Nothing magical, just straight forward sparse matrix construction. I suppose you could bypass the dense matrix
There is a list-of-lists sparse format that looks a bit like your data.
In [356]: Ml=MA.tolil()
In [357]: Ml.rows
Out[357]: array([[], [], [], [0, 1, 2], [], [0, 1], []], dtype=object)
In [358]: Ml.data
Out[358]: array([[], [], [], [1, 2, 1], [], [2, 2], []], dtype=object)
Conceivably you could construct an empty sparse.lil_matrix((n,m)) matrix, and set it's .data attribute directly. But you'd also have to calculate the rows attribute.
You could also look at the data, row. col attributes of the coo format matrix, and decide it would be easy to construct the equivalent from your A list of lists.
One way or other you have to decide how the non-zero rows get padded to the full length.
This:
array = numpy.array ( [ x['feature1'] for x in ver ] )
Or you need to be more clear in your example...
Generally your idea of trying to apply astype to each column is fine.
In [590]: X[:,0].astype(int)
Out[590]: array([1, 2, 3, 4, 5])
But you have to collect the results in a separate list. You can't just put them back in X. That list can then be concatenated.
In [601]: numlist=[]; obj_ind=[]
In [602]: for ind in range(X.shape[1]):
.....: try:
.....: x = X[:,ind].astype(np.float32)
.....: numlist.append(x)
.....: except:
.....: obj_ind.append(ind)
In [603]: numlist
Out[603]: [array([ 3., 4., 5., 6., 7.], dtype=float32)]
In [604]: np.column_stack(numlist)
Out[604]:
array([[ 3.],
[ 4.],
[ 5.],
[ 6.],
[ 7.]], dtype=float32)
In [606]: obj_ind
Out[606]: [1]
X is a numpy array with dtype object:
In [582]: X
Out[582]:
array([[1, 'A'],
[2, 'A'],
[3, 'C'],
[4, 'D'],
[5, 'B']], dtype=object)
You could use the same conversion logic to create a structured array with a mix of int and object fields.
In [616]: ytype=[]
In [617]: for ind in range(X.shape[1]):
try:
x = X[:,ind].astype(np.float32)
ytype.append('i4')
except:
ytype.append('O')
In [618]: ytype
Out[618]: ['i4', 'O']
In [620]: Y=np.zeros(X.shape[0],dtype=','.join(ytype))
In [621]: for i in range(X.shape[1]):
Y[Y.dtype.names[i]] = X[:,i]
In [622]: Y
Out[622]:
array([(3, 'A'), (4, 'A'), (5, 'C'), (6, 'D'), (7, 'B')],
dtype=[('f0', '<i4'), ('f1', 'O')])
Y['f0'] gives the the numeric field.
I think this might help
def func(x):
a = None
try:
a = x.astype(float)
except:
# x.name represents the current index value
# which is column name in this case
obj.append(x.name)
a = x
return a
obj = []
new_df = df.apply(func, axis=0)
This will keep the object columns as such which you can use later.
Note: While using pandas.DataFrame avoid using iteration using loop as this much slower than performing the same operation using apply.
Use val.item() to convert most NumPy values to a native Python type:
import numpy as np
# for example, numpy.float32 -> python float
val = np.float32(0)
pyval = val.item()
print(type(pyval)) # <class 'float'>
# and similar...
type(np.float64(0).item()) # <class 'float'>
type(np.uint32(0).item()) # <class 'int'>
type(np.int16(0).item()) # <class 'int'>
type(np.cfloat(0).item()) # <class 'complex'>
type(np.datetime64(0, 'D').item()) # <class 'datetime.date'>
type(np.datetime64('2001-01-01 00:00:00').item()) # <class 'datetime.datetime'>
type(np.timedelta64(0, 'D').item()) # <class 'datetime.timedelta'>
...
(A related method np.asscalar(val) was deprecated with 1.16, and removed with 1.23).
For the curious, to build a table of conversions of NumPy array scalars for your system:
for name in dir(np):
obj = getattr(np, name)
if hasattr(obj, 'dtype'):
try:
if 'time' in name:
npn = obj(0, 'D')
else:
npn = obj(0)
nat = npn.item()
print('{0} ({1!r}) -> {2}'.format(name, npn.dtype.char, type(nat)))
except:
pass
There are a few NumPy types that have no native Python equivalent on some systems, including: clongdouble, clongfloat, complex192, complex256, float128, longcomplex, longdouble and longfloat. These need to be converted to their nearest NumPy equivalent before using .item().
If you want to convert (numpy.array OR numpy scalar OR native type OR numpy.darray) TO native type you can simply do :
converted_value = getattr(value, "tolist", lambda: value)()
tolist will convert your scalar or array to python native type. The default lambda function takes care of the case where value is already native.
numpy.float is just the regular Python float type. It's not a NumPy dtype. It's almost certainly not what you need:
>>> import numpy
>>> numpy.float is float
True
If you want the dtype NumPy would coerce your scalar to, just make an array and get its dtype:
>>> numpy.array(7.7).dtype
dtype('float64')
If you want the type NumPy uses for scalars of this dtype, access the dtype's type attribute:
>>> numpy.array(7.7).dtype.type
<class 'numpy.float64'>
You could simply use np.float64(original_float), or whatever numpy type you wish to convert your variable to.
For the record, this code works:
val = 7.7
if isinstance(val, float) is True:
val = np.float64(val)
if isinstance(val, np.float64) is True:
print("Success!")
>>>Success!
Hope this helps.
Edit: I just saw @user2357112 supports Monica's comment to your question and it's important to note that effectively np.float acts the same way as float. The implementation I provided is oriented towards special numpy types like np.float32 or np.float64, the one I used in the test code. But if I performed the same test with just np.float this would be the result:
val = 7.7
if isinstance(val, float) is True:
if isinstance(val, np.float) is True:
print("Success!")
>>>Success!
Thus proving that from the interpreter's point of view float and np.float are pretty much the same type.