Nasty little problem... I have been fooling around with this toy example:
>>> arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
>>> arr
array([['one', [1, 2, 3]],
['two', [4, 5, 6]]], dtype=object)
My first guess was:
>>> np.array(arr[:, 1])
array([[1, 2, 3], [4, 5, 6]], dtype=object)
But that keeps the object dtype, so perhaps then:
>>> np.array(arr[:, 1], dtype=float)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.
You can normally work around this doing the following:
>>> np.array(arr[:, 1], dtype=[('', float)]*3).view(float).reshape(-1, 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected a readable buffer object
Not here though, which was kind of puzzling. Apparently it is the fact that the objects in your array are lists that throws this off, as replacing the lists with tuples works:
>>> np.array([tuple(j) for j in arr[:, 1]],
... dtype=[('', float)]*3).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Since there doesn't seem to be any entirely satisfactory solution, the easiest is probably to go with:
>>> np.array(list(arr[:, 1]), dtype=float)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Although that will not be very efficient, probably better to go with something like:
>>> np.fromiter((tuple(j) for j in arr[:, 1]), dtype=[('', float)]*3,
... count=len(arr)).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Nasty little problem... I have been fooling around with this toy example:
>>> arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
>>> arr
array([['one', [1, 2, 3]],
['two', [4, 5, 6]]], dtype=object)
My first guess was:
>>> np.array(arr[:, 1])
array([[1, 2, 3], [4, 5, 6]], dtype=object)
But that keeps the object dtype, so perhaps then:
>>> np.array(arr[:, 1], dtype=float)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.
You can normally work around this doing the following:
>>> np.array(arr[:, 1], dtype=[('', float)]*3).view(float).reshape(-1, 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected a readable buffer object
Not here though, which was kind of puzzling. Apparently it is the fact that the objects in your array are lists that throws this off, as replacing the lists with tuples works:
>>> np.array([tuple(j) for j in arr[:, 1]],
... dtype=[('', float)]*3).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Since there doesn't seem to be any entirely satisfactory solution, the easiest is probably to go with:
>>> np.array(list(arr[:, 1]), dtype=float)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Although that will not be very efficient, probably better to go with something like:
>>> np.fromiter((tuple(j) for j in arr[:, 1]), dtype=[('', float)]*3,
... count=len(arr)).view(float).reshape(-1, 3)
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
Based on Jaime's toy example I think you can do this very simply using np.vstack():
arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
float_arr = np.vstack(arr[:, 1]).astype(np.float)
This will work regardless of whether the 'numeric' elements in your object array are 1D numpy arrays, lists or tuples.
python - How to convert the 'dtype = object' NumPy array into 'dtype = float' array? - Stack Overflow
python - Converting a data structure of dtype=object to numpy array of dtype=float64 - Stack Overflow
python - Convert numpy object type to float type - Stack Overflow
python - How to convert from a dtype('o') to a dtype(float) in numpy? - Stack Overflow
Yet another way
np.frompyfunc(lambda x: x.replace(',',''),1,1)(arr).astype(float)
frompyfunc returns an object dtype array, which is fine in this case. Often I've found that it is 2x faster than than a list comprehension, but here it times about the same as @coldspeed's:
np.array([v.replace(',', '') for v in arr], dtype=np.float32)
That may be because we are starting with an object dtype array. Direct iteration on an object dtype is a bit slower than iteration on a list, but faster than iteration on a regular numpy array. Like a list, the elements of the array are pointers to strings, and don't require the 'unboxing' that a string dtype array would.
(and 2 to 3 x faster than the np.char version).
Simple way to do it is remove every comma:
np.array([v.replace(',', '') for v in arr], dtype=np.float32)
If you have pandas, to_numeric is a good option. It gracefully handles any invalid values that may creep in post replacement.
pd.to_numeric([v.replace(',', '') for v in arr], errors='coerce', downcast='float')
Both methods return a float array as output.
Short answer:
M_rref.astype(float) will fix your problem.
Long answer:
Why does this happen?
In order to find out, try going back to your matrix M and try the following:
>>> print(M[0])
>>> print(type(M[0]))
You will get:
4
sympy.core.numbers.Integer
That's why. Even though it says it's 4, it's actually a SymPy object that is stored inside the SymPy matrices. And that's the way it should be: SymPy is for symbolic mathematics. Therefore, when you convert it to a numpy array, NumPy will recognize sympy.core.numbers.Integer as an object.
Hope that answers your question.
In this PR there is a note about incorrect conversion of numpy matrices to SymPy matrices. Using that (and not float, which I don' think is intended to convert contents of an object to floats) I would do:
>>> sy.Matrix(M.rows, M.cols, [sympify(x) for x in M_rref]).n()
This should give you a SymPy matrix with entries evaluated as floats.
Lets take as the starting point a list of lists or equivalently an object array of lists:
A = [[], [], [], [1,2,1], [], [2, 2, 0, 0], []]
A = array([[], [], [], [1,2,1], [], [2, 2, 0, 0], []], dtype=object)
If the sublists were all the same length, np.array([...]) would give you a 2d array, one row for each sublist, and columns matching their common length. But since they are unequal in length, it can only make it a 1d array, where each element is a pointer to one of these sublists - i.e. dtype=object.
I can imagine 2 ways of constructing a 2d array:
- pad each sublist to a common length
- insert each sublist into an empty array of the appropriate size.
Basically it requires common Python iteration; it's not a common enough task to have a wiz-bang numpy function.
For example:
In [346]: n=len(A)
In [348]: m=max([len(x) for x in A])
In [349]: AA=np.zeros((n,m),int)
In [350]: for i,x in enumerate(A):
.....: AA[i,:len(x)] = x
In [351]: AA
Out[351]:
array([[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[1, 2, 1, 0],
[0, 0, 0, 0],
[2, 2, 0, 0],
[0, 0, 0, 0]])
To get a sparse matrix:
In [352]: from scipy import sparse
In [353]: MA=sparse.coo_matrix(AA)
In [354]: MA
Out[354]:
<7x4 sparse matrix of type '<class 'numpy.int32'>'
with 5 stored elements in COOrdinate format>
Nothing magical, just straight forward sparse matrix construction. I suppose you could bypass the dense matrix
There is a list-of-lists sparse format that looks a bit like your data.
In [356]: Ml=MA.tolil()
In [357]: Ml.rows
Out[357]: array([[], [], [], [0, 1, 2], [], [0, 1], []], dtype=object)
In [358]: Ml.data
Out[358]: array([[], [], [], [1, 2, 1], [], [2, 2], []], dtype=object)
Conceivably you could construct an empty sparse.lil_matrix((n,m)) matrix, and set it's .data attribute directly. But you'd also have to calculate the rows attribute.
You could also look at the data, row. col attributes of the coo format matrix, and decide it would be easy to construct the equivalent from your A list of lists.
One way or other you have to decide how the non-zero rows get padded to the full length.
This:
array = numpy.array ( [ x['feature1'] for x in ver ] )
Or you need to be more clear in your example...
131.199997,1.31199997e+02 are equivalent displays of the same number. They are both 'normal floats'.
In:
array([[131.199997, 137.149994, 131.199997, 136.320007, 117.08859299999999,
615015],
[135.199997, 136.570007, 134.330002, 135.639999, 116.504501, 352835],
[131.419998, 133.5, 130.759995, 131.779999, 113.18906399999999,
209805]], dtype=object)
each element is a Python float, and is formatted individually, regardless of the value. Note how some strings are long, other much short.
In:
a = a.astype('float64') # converting to `float64`
a
array([[ 1.31199997e+02, 1.37149994e+02, 1.31199997e+02,
1.36320007e+02, 1.17088593e+02, 6.15015000e+05],
[ 1.35199997e+02, 1.36570007e+02, 1.34330002e+02,
1.35639999e+02, 1.16504501e+02, 3.52835000e+05],
[ 1.31419998e+02, 1.33500000e+02, 1.30759995e+02,
1.31779999e+02, 1.13189064e+02, 2.09805000e+05]])
the array is displayed as a whole, using a format that will work equally well for the smaller values (1e2, 100) and the larger ones (1e5, 100000). With this format it use neat columns, show the 2d array structure.
While you can control how numpy displays such an array, it does not change the underlying numeric values. And for fast numpy calculations, you want this numeric dtype, not the object one.
Try df.sample(3).values[:,1:-1]. That should just be the float values that are all around 100. It's the last integer column with values like 209805 that's triggering the scientific notation.
Better yet, select the columns from the dataframe, "Open,High,Low,Close,Adj Close" before applying .values. Those are all float64 dtype, and the resulting array will have that dtype as well. Select the integer volume column separately. You are already treating the string/object date column separately.
Try:
a = df[df.columns[1:-1]].sample().values
Try adding:
np.set_printoptions(suppress=True)
As the first line under import numpy as np.
Hi guys, im struggeling with my dataframe.
Im trying to get a percentage of 2 columns, but when I divide the 2 I get this error: unsupported operand type(s) for /: 'float' and 'str'
The dtypes of the columns are an 'object'
Im aware that I have empty data in my columns, but is that an issue?
Would appreciate your help!
1. Using pandas.Series.astype
You can use
pandas.Series.astypeYou can do something like this :
weather["Temp"] = weather.Temp.astype(float)
2. Using pd.to_numeric
You can also use
pd.to_numericthat will convert the column from object to floatFor details on how to use it checkout this link :http://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_numeric.html
Example :
s = pd.Series(['apple', '1.0', '2', -3]) print(pd.to_numeric(s, errors='ignore')) print("=========================") print(pd.to_numeric(s, errors='coerce'))Output:
0 apple 1 1.0 2 2 3 -3 ========================= dtype: object 0 NaN 1 1.0 2 2.0 3 -3.0 dtype: float64In your case you can do something like this:
weather["Temp"] = pd.to_numeric(weather.Temp, errors='coerce')
3. Using convert_objects
Other option is to use
convert_objectsExample is as follows
>> pd.Series([1,2,3,4,'.']).convert_objects(convert_numeric=True) 0 1 1 2 2 3 3 4 4 NaN dtype: float64You can use this as follows:
weather["Temp"] = weather.Temp.convert_objects(convert_numeric=True)I have showed you examples because if any of your column won't have a number then it will be converted to
NaN... so be careful while using it.
I tried all methods suggested here but sadly none worked. Instead, found this to be working:
df['column'] = pd.to_numeric(df['column'],errors = 'coerce')
And then check it using:
print(df.info())