Use std by row (axis=1):
df['stdDev'] = df[['A', 'B', 'C', 'D']].std(axis=1)
output:
Key A B C D stdDev
0 X 1 2 3 4 1.290994
1 y 4 5 6 7 1.290994
2 z 8 9 10 11 1.290994
I think you should be able to figure out how to do this on your own with the public documentation. Anyhow:
import pandas as pd
my_dict = {
"key": ["x", "y", "z"],
"A": [1,2,3],
"B": [4,5,6]
}
df = pd.DataFrame(data=my_dict)
df["std"] = df.std(axis=1)
print(df)
Output:
0 x 1 4 2.12132
1 y 2 5 2.12132
2 z 3 6 2.12132
You can first create a Series out of both columns. Then compute your calculations:
s = pd.concat([df.A, df.B])
s.mean()
s.std()
s.count()
Output
2.5
0.5773502691896257
4
IIUC, select your columns and can use numpy's nanmean and nanstd
cols = ['A', 'B']
np.nanmean(df[cols])
np.nanstd(df[cols])
For the count, use the count() function which already exclude nans
df[cols].count().sum()
This works because nanmean and nanstd (like most numpy methods) have axis=None as default and just consider all values instead of running against a specific axis.
Bear in mind that pandas std() use as default 1 degree of freedom, while numpy uses 0. Depending on which behavior you want, you may specify
np.nanstd(df[cols], ddof=1)
You could convert the dataframe to be a single column with stack (this changes the shape from 5x3 to 15x1) and then take the standard deviation:
df.stack().std() # pandas default degrees of freedom is one
Alternatively, you can use values to convert from a pandas dataframe to a numpy array before taking the standard deviation:
df.values.std(ddof=1) # numpy default degrees of freedom is zero
Unlike pandas, numpy will give the standard deviation of the entire array by default, so there is no need to reshape before taking the standard deviation.
A couple of additional notes:
The numpy approach here is a bit faster than the pandas one, which is generally true when you have the option to accomplish the same thing with either numpy or pandas. The speed difference will depend on the size of your data, but numpy was roughly 10x faster when I tested a few different sized dataframes on my laptop (numpy version 1.15.4 and pandas version 0.23.4).
The numpy and pandas approaches here will not give exactly the same answers, but will be extremely close (identical at several digits of precision). The discrepancy is due to slight differences in implementation behind the scenes that affect how the floating point values get rounded.
Use axis=None
Since pandas 2.0.0, you can use df.mean(axis=None) to compute mean over the entire dataframe. Since pandas 3.0.0, you can use df.std(axis=None) to compute standard deviation over the entire dataframe.
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df.mean(axis=None) # 3.5
df.std(axis=None) # 1.8708286933869707
Note that DataFrame.std sets ddof=1 by default while Numpy's std sets ddof=0 by default. You can check the relationships as follows:
df.std(axis=None, ddof=0) == df.values.std() # True
df.std(axis=None) == df.values.std(ddof=1) # True
Good thing about pandas mean and std is that it ignores NaN values for you if the dataframe has any whereas with numpy, you have to explicitly filter NaNs out.
# a dataframe with a NaN value
df = pd.DataFrame({'A': [1, float("nan"), 3], 'B': [4, 5, 6]})
df.values.mean() # nan <--- numpy mean/std becomes meaningless
df.values.std() # nan
df.mean(axis=None) # 3.8 <--- pandas mean/std ignores NaNs
df.std(axis=None) # 1.9235384061671346