Not exactly what was asked in the question, but if you wanted to avoid NaN values, calculate the population standard deviation, specified with std(ddof=0):

>>> print(df.groupby('Category').std(ddof=0))
                 A         B         C         D         E         F
Category                                                            
A         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
B         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
C         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
D         0.248192  0.195198  0.275101  0.194955  0.190215  0.052423
E         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
F         0.288417  0.127854  0.065012  0.110096  0.354885  0.191643
G         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
H         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000

Note the different defaults for ddof (Delta Degrees of Freedom):

  • Pandas: DataFrame.std has default ddof=1 for sample standard deviation (divisor: N − 1)
  • NumPy: numpy.std has default ddof=0 for population standard deviation (divisor: N)
Answer from Mike T on Stack Overflow
Top answer
1 of 2
25

Not exactly what was asked in the question, but if you wanted to avoid NaN values, calculate the population standard deviation, specified with std(ddof=0):

>>> print(df.groupby('Category').std(ddof=0))
                 A         B         C         D         E         F
Category                                                            
A         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
B         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
C         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
D         0.248192  0.195198  0.275101  0.194955  0.190215  0.052423
E         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
F         0.288417  0.127854  0.065012  0.110096  0.354885  0.191643
G         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
H         0.000000  0.000000  0.000000  0.000000  0.000000  0.000000

Note the different defaults for ddof (Delta Degrees of Freedom):

  • Pandas: DataFrame.std has default ddof=1 for sample standard deviation (divisor: N − 1)
  • NumPy: numpy.std has default ddof=0 for population standard deviation (divisor: N)
2 of 2
5

You could fillna to replace the missing values - passing in a DataFrame with the last value of each group.

In [86]: (df.groupby('Category').std()
    ...:    .fillna(df.groupby('Category').last()))

Out[86]: 
                 A         B         C         D         E         F
Category                                                            
A         0.500200  0.791039  0.498083  0.360320  0.965992  0.537068
B         0.714371  0.636975  0.153347  0.936872  0.000649  0.692558
C         0.295330  0.638823  0.133570  0.272600  0.647285  0.737942
D         0.350996  0.276052  0.389051  0.275708  0.269005  0.074137
E         0.639271  0.486151  0.860172  0.870838  0.831571  0.404813
F         0.407883  0.180813  0.091941  0.155699  0.501884  0.271024
G         0.384157  0.858391  0.278563  0.677627  0.998458  0.829019
H         0.109465  0.085861  0.440557  0.925500  0.767791  0.626924
🌐
GitHub
github.com › pandas-dev › pandas › issues › 21786
.rolling().std() only returns NaN in Python3.7 · Issue #21786 · pandas-dev/pandas
July 7, 2018 - import pandas as pd d = {"col": [1, 23, 231, 231, 4, 353, 62, 3, 56, 43, 354, 43, 231, 21, 7]} df = pd.DataFrame(data=d) std = df["col"].std() df["mean5"] = df["col"].rolling(5).mean() df["std5"] = df["col"].rolling(5).std() print(std) print(df[["mean5", "std5"]]) # OUTPUT 130.20855066648528 mean5 std5 0 NaN NaN 1 NaN NaN 2 NaN NaN 3 NaN NaN 4 98.0 NaN 5 168.4 NaN 6 176.2 NaN 7 130.6 NaN 8 95.6 NaN 9 103.4 NaN 10 103.6 NaN 11 99.8 NaN 12 145.4 NaN 13 138.4 NaN 14 131.2 NaN ·
Author: pandas-dev
🌐
NumPy
numpy.org › doc › stable › reference › generated › numpy.nanstd.html
numpy.nanstd — NumPy v2.5 Manual
New in version 2.0.0. ... If out is None, return a new array containing the standard deviation, otherwise return a reference to the output array. If ddof is >= the number of non-NaN elements in a slice or the slice contains only NaNs, then the result for that slice is NaN.
🌐
GitHub
github.com › dask › dask › issues › 5725
std function returns NaN for long floats · Issue #5725 · dask/dask
December 18, 2019 - Calling .groupby().std() on a dataframe with long floating point numbers returns NaNs and a warning. This occurs because internally the variance is calculated as a (small) negative number. Seems to be a resurfacing of #4233 import pandas...
Author: dask
🌐
Medium
medium.com › @amit25173 › understanding-pandas-dataframe-std-90f742cc9d3a
Understanding pandas.DataFrame.std() | by Amit Yadav | Medium
March 6, 2025 - When skipna=False, any presence of NaN results in NaN for the standard deviation. Basically, pandas says, “I can’t compute this with incomplete data.” ... By default, pandas std() uses ddof=1, which calculates the sample standard deviation.
🌐
GeeksforGeeks
geeksforgeeks.org › python › python-pandas-dataframe-std
Python | Pandas dataframe.std() - GeeksforGeeks
October 22, 2019 - We are going to set skipna to be true. If we do not skip the NaN values then it will result in NaN values. ... # importing pandas as pd import pandas as pd # Creating the dataframe df = pd.read_csv("nba.csv") # STD over the column axis.
🌐
Stack Overflow
stackoverflow.com › questions › 60918179 › panda-dataframe-mean-std-dev-gives-nan-although-no-nan-values
python - panda dataframe - mean & std dev gives nan although no nan values - Stack Overflow
In the screenshot below, the two first False are from isna() queries. The two nan values are from mean and std and the dataframe below is clearly cause it's tried to normalise against these values. python · pandas · dataframe · mean · Share · Share a link to this question ·
Find elsewhere
🌐
Medium
medium.com › @datasci-rahul › pandas-numpy-return-different-values-of-standard-deviation-8aea4cd40db1
Pandas & NumPy return different values of standard deviation! | by Rahul Sharma | Medium
February 3, 2023 - In Pandas, missing values are represented as NaN (Not a Number), and by default, Pandas uses the ddof (degrees of freedom) value of 1 when calculating the standard deviation.
🌐
Vultr Docs
docs.vultr.com › python › third-party › pandas › DataFrame › std
Python Pandas DataFrame std() - Calculate Standard Deviation | Vultr Docs
December 24, 2024 - With the std() function, any NaN or NA values are automatically ignored, ensuring accurate statistical calculations.
🌐
Programiz
programiz.com › python-programming › pandas › methods › std
Pandas std()
import pandas as pd data = {'A': ... = df.std(skipna=True) print(std_dev_skipna) ... Here, by setting skipna=True, the function skips over any NaN values present in the data when calculating the standard deviation....
🌐
Statology
statology.org › home › how to calculate standard deviation in pandas (with examples)
How to Calculate Standard Deviation in Pandas (With Examples)
September 27, 2021 - You can use the following methods ... of One Column ... Note that the std() function will automatically ignore any NaN values in the DataFrame when calculating the standard deviation....
🌐
GitHub
github.com › pandas-dev › pandas › issues › 11524
BUG: pandas std broken, erratic behavior · Issue #11524 · pandas-dev/pandas
November 5, 2015 - # NaN appears if there are enough decimal places after the comma print(df.groupby(level=0).agg([np.mean, np.std])) # 0 # mean std #uid #15 538512.198638 NaN df['bla'] = 911. print(df.groupby(level=0).agg([np.mean, np.std])) # 0 bla # mean std ...
Author: pandas-dev
🌐
Spark By {Examples}
sparkbyexamples.com › home › pandas › pandas dataframe std() method
Pandas DataFrame std() Method - Spark By {Examples}
December 6, 2024 - In Pandas, the std() method is used to calculate the standard deviation of the values in a DataFrame or a Series. The standard deviation measures the
🌐
Finxter
blog.finxter.com › pandas-nan
Pandas NaN — Working With Missing Data – Be on the Right Side of Change
November 4, 2020 - When you restrict the columns only to price, no rows will be dropped, because no NaN value is present. Problem: What happens to indices after dropping certain rows? import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df" # ---------- # make fuel aspiration body-style price engine-size # 0 audi gas turbo sedan 30000 2.0 # 1 dodge gas std sedan 17000 1.8 # 2 mazda diesel std sedan 17000 NaN # 3 porsche gas turbo convertible 120000 6.0 # 4 volvo diesel std sedan 25000 2.0 # ---------- df.drop([0, 1, 2], inplace=True) df.reset_index(inplace=True) result = df.index.to_list() print(result) # [0, 1]
🌐
GitHub
github.com › pandas-dev › pandas › issues › 1884
pandas.rolling_std() first value is nan · Issue #1884 · pandas-dev/pandas
September 10, 2012 - The window is 3, but we want a std at min_periods=1. The one-period standard deviation is trivially 0. In [28]: pandas.rolling_std(np.array([1,2,3,4,5], dtype='double'), 3, min_periods=1) Out[28]: array([ nan, 0.70710678, 1. , 1. , 1. ])...
Author: pandas-dev
🌐
Pandas
pandas.pydata.org › docs › reference › api › pandas.DataFrame.describe.html
pandas.DataFrame.describe — pandas 3.0.6 documentation
>>> df.describe(exclude=[object]) categorical numeric count 3 3.0 unique 3 NaN top f NaN freq 1 NaN mean NaN 2.0 std NaN 1.0 min NaN 1.0 25% NaN 1.5 50% NaN 2.0 75% NaN 2.5 max NaN 3.0
🌐
University of Texas at Austin
het.as.utexas.edu › HET › Software › Numpy › reference › generated › numpy.nanstd.html
numpy.nanstd — NumPy v1.9 Manual
For all-NaN slices or slices with zero degrees of freedom, NaN is returned and a RuntimeWarning is raised. New in version 1.8.0. ... The standard deviation is the square root of the average of the squared deviations from the mean: std = sqrt(mean(abs(x - x.mean())**2)).