Prometheus doesn't provide functionality, which can be used for returning cumulative increase over multiple time series on the selected time range.
If you still need this functionality, then try VictoriaMetrics - Prometheus-like monitoring solution I work on. It allows calculating cumulative increase over multiple counters. For example, the following MetricsQL query returns cumulative increase over all the time series with http_requests_total name on the selected time range in Grafana:
running_sum(sum(increase(http_requests_total)))
How does it work?
It calculates increase per each time series with the
http_requests_totalname. Note that theincrease()in the query above doesn't contain lookbehind window in square brackets. VictoriaMetrics automatically sets the lookbehind window to thestepvalue, which is passed by Grafana to /api/v1/query_range endpoint. Thestepvalue is the interval between points on the graph.It sums increases returned at step 1 with the sum() function individually per each point on the graph.
It calculates cumulative increase over per-
stepincreases returned at step 2 with the running_sum function.
Prometheus doesn't provide functionality, which can be used for returning cumulative increase over multiple time series on the selected time range.
If you still need this functionality, then try VictoriaMetrics - Prometheus-like monitoring solution I work on. It allows calculating cumulative increase over multiple counters. For example, the following MetricsQL query returns cumulative increase over all the time series with http_requests_total name on the selected time range in Grafana:
running_sum(sum(increase(http_requests_total)))
How does it work?
It calculates increase per each time series with the
http_requests_totalname. Note that theincrease()in the query above doesn't contain lookbehind window in square brackets. VictoriaMetrics automatically sets the lookbehind window to thestepvalue, which is passed by Grafana to /api/v1/query_range endpoint. Thestepvalue is the interval between points on the graph.It sums increases returned at step 1 with the sum() function individually per each point on the graph.
It calculates cumulative increase over per-
stepincreases returned at step 2 with the running_sum function.
Using sum(increase(http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"}[$__interval])) as query is a good starting point and that's as far as Prometheus can go.
However, in order to get the cumulative graph, you need to leverage a "Transformation" in the Grafana UI as I'll describe below with a concrete example.
See for example this graph in the Grafana Playground
Let's select a single timeseries upper_95 that we want to plot the cumulative sum for:
Then select the "Transformations" tab and click "Add Transformation"
Choose the "Add field from calculation" transformation
Then choose Mode "Cumulative functions" and Calculation "Total" (should be the default). The documentation says:
Cumulative functions - Apply functions on the current row and all preceding rows. Total - Calculates the cumulative total up to and including the current row.
which is exactly what we want.
Since we have multiple timeseries we need to also select "upper_95" as Field, but if you only have one you can leave it as blank.
Now you have the cumulative sum graph available and you can select it in the legend to hide the other ones:
prometheus - Grafana - Promql cumulative graph - Stack Overflow
Help with PromQL query (sum over time)
Display sum of all gauge's values
How can I make a cumulative sum graph in grafana, from an elasticsearch data source? - Stack Overflow
What you need is the increase() function, that will calculate the difference between the counter values at the start and at the end of the specified time interval. It also correctly handles counter resets during that time period (if any).
increase(http_requests_total[24h])
If you have multiple counters http_requests_total (e.g. from multiple instances) and you need to get the cumulative count of requests, use the sum() operator:
sum(increase(http_requests_total[24h]))
See also my answer to that part of the question about using Grafana's time range selection in queries.
SO won't let me comment on Yoory's answer so I have to make a new one...
In Grafana 5.3, they introduced $__range for Prometheus that's easier to use:
sum(rate(http_requests_total[$__range]))
This variable represents the range for the current dashboard. It is calculated by to - from
http://docs.grafana.org/features/datasources/prometheus/
PromQL in Prometheus doesn't support functions for cumulative calculations. If you need cumulative calculations, then try VictoriaMetrics instead - its query language - MetricsQL - provides running_sum() function.
P.s.: I'm the author of VictoriaMetrics.
The problem is that sum_over_time returns a single value which is the total sum of all data points in the interval, whereas what you want is - for each timestamp - the sum up to that timestamp. That is unfortunately not possible in Prometheus alone.
But Grafana has support for that using an "Add field from calculation" transformation using "Cumulative functions" as mode . See https://stackoverflow.com/a/79453819/574351
Hello,
I have this graph monitoring the bandwidth of a VLAN on a switch every 1m using SNMP Exporter, but I also what to get the total/sum data over time, so if I select the last hour it will show x amount inbound and x amount outbound.
sum by(ifName) (irate(ifHCInOctets{instance=~"192.168.200.10", job="snmp_exporter", ifName=~".*(1001).*"}[1m])) * 8My current graph:
I'd like to duplicate and create a stat panel show how much data in total has passed over what period I choose that's all.
For the metric I'm not sure whether to use bytes(SI) or bytes(IEC), but are similar if I change to either.
Not sure how to calculate this, but I have this created for the past 1 hour.
by copying the PromQL in Grafana and changing to a stat panel and then editing to use this:
Not sure if this is ok as I'm not sure how to calculate it all, maths was never my best subject.
Any help would be great.
I think something like is close: with sum_over_time
sum by(ifName) (sum_over_time(ifHCInOctets{instance=~"192.168.200.10", job="snmp_exporter", ifName=~".*(1001).*"}[1m])) * 8but it comes back as 85.8 Pib when it should be 85.8 TB with my calculations.
EDIT
Observium:
What Grafana shows
After contacting grafana directly I received the answer that this is not currently possible. All such computation must be done by the data source, not by grafana.
Not exactly an answer to the original question but since the answer is "It's impossible" I thought I'd share my work-around for this missing feature. As this is what I did instead.
Here is how to do it in MariaDB:
SET @cumul := 0;
SELECT
`seq`
, @cumul := @cumul + `seq` AS `cumul`
FROM seq_1_to_20;
The data has to be ordered in sequence for it to add up correctly.
to explain it:
- set a variable called @cumul
- add the value you want to make cumulative and select it as a column
EDIT:
In Grafana it does not let you run 2 queries at the same time and it also makes use of Connection Pooling, meaning it speeds up the system by running queries over different connections, so if you run them as 2 queries the variable is not reset.
After hacking at it, this version works in Grafana for MariaDB (and probably MySQL too)
SELECT
`seq`
, @cumul := @cumul + `seq` AS `cumul`
FROM seq_1_to_20, (SELECT @cumul:= 0) b