If you know, given a series, how to compute the semi std - then you use .rolling().apply() with that function.
Using the definition from https://www.investopedia.com/terms/s/semideviation.asp
We cannot use the built-in std because we need to use the whole group average, but only compute deviations based on those observations in the group that are below the average.
Looks like we'll need to define the semi-deviation as a function:
import numpy as np
def semi_std(ser):
average = np.nanmean(ser)
r_below = ser[ser < average]
return np.sqrt(1/len(r_below) * np.sum((average - r_below)**2))
x.rolling(window).apply(semi_std, raw=True)
# using raw=True speeds up the computation - this is applicable
# if the function works well with a numpy array instead of a Series
# also, if possible, investigate using numba.
I think for maximum robustness, probably add a condition that if len(r_below) == 0 then the result is 0 or NaN (it's a matter of definition - but 0 is probably the reasonable choice).
Pandas
pandas.pydata.org › docs › reference › api › pandas.api.typing.Rolling.sem.html
pandas.api.typing.Rolling.sem — pandas 3.0.5 documentation
Aggregating sem for DataFrame. ... A minimum of one period is required for the calculation. ... >>> s = pd.Series([0, 1, 2, 3]) >>> s.rolling(2, min_periods=1).sem() 0 NaN 1 0.5 2 0.5 3 0.5 dtype: float64
Pandas
pandas.pydata.org › pandas-docs › stable › reference › api › pandas.core.window.rolling.Rolling.sem.html
pandas.core.window.rolling.Rolling.sem — pandas 2.3.3 documentation
Calling rolling with DataFrames. pandas.Series.sem · Aggregating sem for Series. pandas.DataFrame.sem · Aggregating sem for DataFrame. Notes · A minimum of one period is required for the calculation. Examples · >>> s = pd.Series([0, 1, 2, 3]) >>> s.rolling(2, min_periods=1).sem() 0 NaN 1 0.707107 2 0.707107 3 0.707107 dtype: float64 ·
29:42
12 Practical Pandas Rolling Examples (You’ll Actually Use!) - ...
Calmcode - pandas datetime: Rolling Stats
Pandas Windows Rolling Function Tutorial
How to Use Pandas Rolling - A Simple Illustrated Guide
09:35
Moving Average (Rolling Average) in Pandas and Python - Set Window ...
04:05
Calculation of mean with rolling lookback window in pandas DataFrame ...
Pandas
pandas.pydata.org › docs › dev › reference › api › pandas.core.window.rolling.Rolling.sem.html
pandas.core.window.rolling.Rolling.sem — pandas 3.0.0.dev0+1840.ga4e814954b documentation
Aggregating sem for DataFrame. ... A minimum of one period is required for the calculation. ... >>> s = pd.Series([0, 1, 2, 3]) >>> s.rolling(2, min_periods=1).sem() 0 NaN 1 0.707107 2 0.707107 3 0.707107 dtype: float64
Snowflake Documentation
docs.snowflake.com › en › developer-guide › snowpark › reference › python › 1.22.1 › modin › pandas_api › snowflake.snowpark.modin.pandas.window.Rolling.sem
modin.pandas.Rolling.sem | Snowflake Documentation
Rolling.sem(ddof: int = 1, numeric_only: bool = False, *args: Any, **kwargs: Any)[source]¶
Pandas
pandas.pydata.org › pandas-docs › version › 2.0 › reference › api › pandas.core.window.rolling.Rolling.sem.html
pandas.core.window.rolling.Rolling.sem — pandas 2.0.3 documentation
Calling rolling with DataFrames. pandas.Series.sem · Aggregating sem for Series. pandas.DataFrame.sem · Aggregating sem for DataFrame. Notes · A minimum of one period is required for the calculation. Examples · >>> s = pd.Series([0, 1, 2, 3]) >>> s.rolling(2, min_periods=1).sem() 0 NaN 1 0.707107 2 0.707107 3 0.707107 dtype: float64 ·
GitHub
github.com › pandas-dev › pandas › issues › 63180
BUG: Wrong Results for `Rolling.sem` · Issue #63180 · pandas-dev/pandas
November 23, 2025 - The rolling standard error of the mean (SEM) produces wrong values compared to the non-rolling version Series.sem().
Author: pandas-dev
Top answer 1 of 3
1
If you know, given a series, how to compute the semi std - then you use .rolling().apply() with that function.
Using the definition from https://www.investopedia.com/terms/s/semideviation.asp
We cannot use the built-in std because we need to use the whole group average, but only compute deviations based on those observations in the group that are below the average.
Looks like we'll need to define the semi-deviation as a function:
import numpy as np
def semi_std(ser):
average = np.nanmean(ser)
r_below = ser[ser < average]
return np.sqrt(1/len(r_below) * np.sum((average - r_below)**2))
x.rolling(window).apply(semi_std, raw=True)
# using raw=True speeds up the computation - this is applicable
# if the function works well with a numpy array instead of a Series
# also, if possible, investigate using numba.
I think for maximum robustness, probably add a condition that if len(r_below) == 0 then the result is 0 or NaN (it's a matter of definition - but 0 is probably the reasonable choice).
2 of 3
1
The solution is actually pretty simple if one knows how rolling windows work in Pandas. The trick here is to use the shift function on the dataframe first which shifts each row by one position so it becomes the last element of the previous row. Then we can do our rolling mean calculation without having any issues with type conversion etc.
import pandas as pd
from datetime import timedelta
df = pd.DataFrame(data=np.random.randn(5, 4), index=pd.date_range('1/1/2000', periods=5))
df.index = df.index + timedelta(days=10)
print(df)
GitHub
github.com › pandas-dev › pandas › blob › main › pandas › core › window › rolling.py
pandas/pandas/core/window/rolling.py at main · pandas-dev/pandas
window_func = window_aggregations.roll_skew · return self._apply( window_func, name="skew", numeric_only=numeric_only, ) · def sem(self, ddof: int = 1, numeric_only: bool = False): # Raise here so error message says sem instead of std ·
Author: pandas-dev
GeeksforGeeks
geeksforgeeks.org › python-pandas-series-sem
Python | Pandas Series.sem() | GeeksforGeeks
February 5, 2019 - Pandas Series.sem() function return unbiased standard error of the mean over requested axis. The result is normalized by N-1 by default.
GitHub
github.com › tamnd › firepanda › issues › 461
Rolling and expanding var, std and sem, which need a carried spread rather than a carried total · Issue #461 · tamnd/firepanda
2 weeks ago - Measured on pandas 3.0.5, pd.Series([1.0, inf, 2.0, 3.0, 4.0]).rolling(2).var() is [nan, nan, nan, 0.5, 0.5]. A window holding an infinity answers NaN and the windows after it recover, which is both what pandas does and what the arithmetic says, because the variance of a set holding an infinity is genuinely undefined. So unlike the total and the extremes in #454, there is no divergence to register here. Worth measuring rather than assuming. ddof is the first positional parameter of all three, defaulting to one, and it is the first window parameter that belongs to the reduction rather than to the window. Rolling.sem takes it and takes no engine or engine_kwargs, where Rolling.std and Rolling.var take both, so the generator's window table stops being five rows of the same shape.
Author: tamnd
Pandas
pandas.pydata.org › docs › reference › api › pandas.DataFrame.rolling.html
pandas.DataFrame.rolling — pandas 3.0.6 documentation
Execute the rolling operation per single column or row ('single') or over the entire object ('table'). This argument is only implemented when specifying engine='numba' in the method call. Returns: pandas.api.typing.Window or pandas.api.typing.Rolling · An instance of Window is returned if win_type is passed.
Medium
medium.com › @whyamit101 › understanding-pandas-rolling-f8f6d6796c07
Understanding Pandas Rolling. If you think you need to spend $2,000… | by why amit | Medium
February 26, 2025 - You might have heard the term “rolling” tossed around in the context of data analysis, but what does it actually mean? In Pandas, the rolling() function allows you to perform window-based calculations on your data.
Spark By {Examples}
sparkbyexamples.com › home › pandas › pandas rolling() mean, average, sum examples
Pandas rolling() Mean, Average, Sum Examples - Spark By {Examples}
October 14, 2024 - pandas.DataFrame.rolling() function can be used to get the rolling mean, average, sum, median, max, min e.t.c for one or multiple columns. Rolling mean is
Programiz
programiz.com › python-programming › pandas › methods › rolling
Pandas rolling()
The rolling() method returns an object, which is not a final computed result but rather an intermediate object that allows us to apply various aggregation functions within the rolling window. import pandas as pd # create a DataFrame with sequential data data = pd.DataFrame({'value': [1, 2, 3, 4, 5, 6, 7, 8, 9]})
Statology
statology.org › home › how to calculate a rolling mean in pandas
How to Calculate a Rolling Mean in Pandas
October 16, 2023 - This tutorial explains how to calculate a rolling mean for one or more columns in a pandas DataFrame, including examples.
GeeksforGeeks
geeksforgeeks.org › python-pandas-dataframe-rolling
Python | Pandas dataframe.rolling() - GeeksforGeeks
February 21, 2022 - Python is a great language for ... importing and analyzing data much easier. Pandas dataframe.rolling() function provides the feature of rolling window calculations....
Top answer 1 of 3
99
To assign a column, you can create a rolling object based on your Series:
df['new_col'] = data['column'].rolling(5).mean()
The answer posted by ac2001 is not the most performant way of doing this. He is calculating a rolling mean on every column in the dataframe, then he is assigning the "ma" column using the "pop" column. The first method of the following is much more efficient:
%timeit df['ma'] = data['pop'].rolling(5).mean()
%timeit df['ma_2'] = data.rolling(5).mean()['pop']
1000 loops, best of 3: 497 µs per loop
100 loops, best of 3: 2.6 ms per loop
I would not recommend using the second method unless you need to store computed rolling means on all other columns.
2 of 3
14
Edit: pd.rolling_mean is deprecated in pandas and will be removed in future. Instead: Using pd.rolling you can do:
df['MA'] = df['pop'].rolling(window=5,center=False).mean()
for a dataframe df:
Date stock pop
0 2016-01-04 325.316 82
1 2016-01-11 320.036 83
2 2016-01-18 299.169 79
3 2016-01-25 296.579 84
4 2016-02-01 295.334 82
5 2016-02-08 309.777 81
6 2016-02-15 317.397 75
7 2016-02-22 328.005 80
8 2016-02-29 315.504 81
9 2016-03-07 328.802 81
To get:
Date stock pop MA
0 2016-01-04 325.316 82 NaN
1 2016-01-11 320.036 83 NaN
2 2016-01-18 299.169 79 NaN
3 2016-01-25 296.579 84 NaN
4 2016-02-01 295.334 82 82.0
5 2016-02-08 309.777 81 81.8
6 2016-02-15 317.397 75 80.2
7 2016-02-22 328.005 80 80.4
8 2016-02-29 315.504 81 79.8
9 2016-03-07 328.802 81 79.6
Documentation: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rolling.html
Old: Although it is deprecated you can use:
df['MA']=pd.rolling_mean(df['pop'], window=5)
to get:
Date stock pop MA
0 2016-01-04 325.316 82 NaN
1 2016-01-11 320.036 83 NaN
2 2016-01-18 299.169 79 NaN
3 2016-01-25 296.579 84 NaN
4 2016-02-01 295.334 82 82.0
5 2016-02-08 309.777 81 81.8
6 2016-02-15 317.397 75 80.2
7 2016-02-22 328.005 80 80.4
8 2016-02-29 315.504 81 79.8
9 2016-03-07 328.802 81 79.6
Documentation: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.rolling_mean.html
Stack Overflow
stackoverflow.com › questions › 48499815 › sequential-pandas-rolling-data-processing
python 3.6 - sequential pandas rolling data processing - Stack Overflow
January 29, 2018 - If values in the rolling window are above a distinct threshold, rolling values are replaced by the means which works great outside pd.rolling... ... import pandas as pd import numpy as np # Create dummy data df = pd.DataFrame(np.random.randint(0,800,size=(1000, 3)), columns=list('ABC')) # To include this data into the dataframe with rolling means, start by creating a copy df_complete = df.copy() # Use the set of considered window sizes in this loop for ws in [51, 45, 55]: r = df.rolling(window=ws, center=False).mean() # Give the following names to the columns with rolling windows: X_S, # where X - name of data column and S - current window size r.columns = ["%s_%d" % (c, ws) for c in r.columns] # Add new columns to the aggregate dataframe (align using index) df_complete = pd.concat([df_complete, r], axis=1) print(df_complete.sample(5))
Pandas
pandas.pydata.org › pandas-docs › stable › generated › pandas.DataFrame.rolling.html
pandas.DataFrame.rolling — pandas 2.3.3 documentation
The page has been moved to this page
Spark By {Examples}
sparkbyexamples.com › home › pandas › pandas rolling sum
Pandas Rolling Sum - Spark By {Examples}
March 27, 2024 - Pandas DataFrame.rolling(n).sum() function is used to get the sum of rolling windows over a DataFrame. Using this function we can get the rolling sum for