Consider a small 2d array:
In [180]: A=np.arange(12).reshape(3,4)
In [181]: A
Out[181]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
Sum across rows; the result is a (3,) array
In [182]: A.sum(axis=1)
Out[182]: array([ 6, 22, 38])
But to sum (or divide) A by the sum requires reshaping
In [183]: A-A.sum(axis=1)
...
ValueError: operands could not be broadcast together with shapes (3,4) (3,)
In [184]: A-A.sum(axis=1)[:,None] # turn sum into (3,1)
Out[184]:
array([[ -6, -5, -4, -3],
[-18, -17, -16, -15],
[-30, -29, -28, -27]])
If I use keepdims, "the result will broadcast correctly against" A.
In [185]: A.sum(axis=1, keepdims=True) # (3,1) array
Out[185]:
array([[ 6],
[22],
[38]])
In [186]: A-A.sum(axis=1, keepdims=True)
Out[186]:
array([[ -6, -5, -4, -3],
[-18, -17, -16, -15],
[-30, -29, -28, -27]])
If I sum the other way, I don't need the keepdims. Broadcasting this sum is automatic: A.sum(axis=0)[None,:]. But there's no harm in using keepdims.
In [190]: A.sum(axis=0)
Out[190]: array([12, 15, 18, 21]) # (4,)
In [191]: A-A.sum(axis=0)
Out[191]:
array([[-12, -14, -16, -18],
[ -8, -10, -12, -14],
[ -4, -6, -8, -10]])
If you prefer, these actions might make more sense with np.mean, normalizing the array over columns or rows. In any case it can simplify further math between the original array and the sum/mean.
You can keep the dimension with "keepdims=True" if you sum a matrix
For example:
import numpy as np
x = np.array([[1,2,3],[4,5,6]])
x.shape
# (2, 3)
np.sum(x, keepdims=True).shape
# (1, 1)
np.sum(x, keepdims=True)
# array([[21]]) <---the reault is still a 1x1 array
np.sum(x, keepdims=False).shape
# ()
np.sum(x, keepdims=False)
# 21 <--- the result is an integer with no dimesion
The first that comes in mind is to pass a dtype=np.float64 param.
df.sum(axis=1,dtype=np.float64)
However this returns a ValueError:
ValueError: the 'dtype' parameter is not supported in the pandas implementation of sum()
Possible workaround:
Use np.sum(), the underlying library to pandas, instead and pass dtype.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'col1': [35000.0, 35000.0],
'col2': [35000.0, 35000.0]
})
df['col1'] = df['col1'].astype(np.float16)
df['col2'] = df['col2'].astype(np.float16)
#print(df.sum(axis=1)) # --> results in inf
#print(df.sum(axis=1,dtype=np.float64)) # --> results in error message
print(np.sum(df.values, dtype=np.float64, axis=1)) # --> works
There is no solution so far , possible workaround may be as @Anton vBR. However there is already a bug with this When running reductions on dataframe columns of dtype float16, it into a surprising behaviour:
[Already a Bug opened for this on github[(https://github.com/pandas-dev/pandas/issues/22841)
@Ney @hpaulj is correct, you need to experiment, but I suspect you don't realize that summation for some arrays can occur along axes. Observe the following which reading the documentation
>>> a
array([[0, 0, 0],
[0, 1, 0],
[0, 2, 0],
[1, 0, 0],
[1, 1, 0]])
>>> np.sum(a, keepdims=True)
array([[6]])
>>> np.sum(a, keepdims=False)
6
>>> np.sum(a, axis=1, keepdims=True)
array([[0],
[1],
[2],
[1],
[2]])
>>> np.sum(a, axis=1, keepdims=False)
array([0, 1, 2, 1, 2])
>>> np.sum(a, axis=0, keepdims=True)
array([[2, 4, 0]])
>>> np.sum(a, axis=0, keepdims=False)
array([2, 4, 0])
You will notice that if you don't specify an axis (1st two examples), the numerical result is the same, but the keepdims = True returned a 2D array with the number 6, whereas, the second incarnation returned a scalar.
Similarly, when summing along axis 1 (across rows), a 2D array is returned again when keepdims = True.
The last example, along axis 0 (down columns), shows a similar characteristic... dimensions are kept when keepdims = True.
Studying axes and their properties is critical to a full understanding of the power of NumPy when dealing with multidimensional data.
An example showing keepdims in action when working with higher dimensional arrays. Let's see how the shape of the array changes as we do different reductions:
import numpy as np
a = np.random.rand(2,3,4)
a.shape
# => (2, 3, 4)
# Note: axis=0 refers to the first dimension of size 2
# axis=1 refers to the second dimension of size 3
# axis=2 refers to the third dimension of size 4
a.sum(axis=0).shape
# => (3, 4)
# Simple sum over the first dimension, we "lose" that dimension
# because we did an aggregation (sum) over it
a.sum(axis=0, keepdims=True).shape
# => (1, 3, 4)
# Same sum over the first dimension, but instead of "loosing" that
# dimension, it becomes 1.
a.sum(axis=(0,2)).shape
# => (3,)
# Here we "lose" two dimensions
a.sum(axis=(0,2), keepdims=True).shape
# => (1, 3, 1)
# Here the two dimensions become 1 respectively