This works for me:
sum(increase(http_request_duration_seconds_count{ecs_cluster=~"$ecs_cluster", instance_id=~"$instance_id"}[$__range]))
Activated instant query and set calculation to last not null
Here is the pane JSON:
{
"cacheTimeout": null,
"datasource": "Prometheus",
"description": "",
"fieldConfig": {
"defaults": {
"custom": {},
"unit": " requests",
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
},
"mappings": [],
"nullValueMode": "connected"
},
"overrides": []
},
"gridPos": {
"h": 2,
"w": 5,
"x": 0,
"y": 4
},
"id": 4,
"interval": null,
"links": [],
"maxDataPoints": 100,
"options": {
"reduceOptions": {
"values": false,
"calcs": [
"lastNotNull"
],
"fields": ""
},
"orientation": "horizontal",
"textMode": "auto",
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"fieldOptions": {
"calcs": [
"lastNotNull"
]
}
},
"pluginVersion": "7.1.0",
"targets": [
{
"expr": "sum(increase(http_request_duration_seconds_count{ecs_cluster=~\"$ecs_cluster\", instance_id=~\"$instance_id\"}[$__range]))",
"hide": false,
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "",
"type": "stat"
}
time series - Grafana Prometheus Counter - Stack Overflow
Aggregating counters over time
How to create graph of counter-changes
Properly plot counters with Prometheus in Grafana
This works for me:
sum(increase(http_request_duration_seconds_count{ecs_cluster=~"$ecs_cluster", instance_id=~"$instance_id"}[$__range]))
Activated instant query and set calculation to last not null
Here is the pane JSON:
{
"cacheTimeout": null,
"datasource": "Prometheus",
"description": "",
"fieldConfig": {
"defaults": {
"custom": {},
"unit": " requests",
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
},
"mappings": [],
"nullValueMode": "connected"
},
"overrides": []
},
"gridPos": {
"h": 2,
"w": 5,
"x": 0,
"y": 4
},
"id": 4,
"interval": null,
"links": [],
"maxDataPoints": 100,
"options": {
"reduceOptions": {
"values": false,
"calcs": [
"lastNotNull"
],
"fields": ""
},
"orientation": "horizontal",
"textMode": "auto",
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"fieldOptions": {
"calcs": [
"lastNotNull"
]
}
},
"pluginVersion": "7.1.0",
"targets": [
{
"expr": "sum(increase(http_request_duration_seconds_count{ecs_cluster=~\"$ecs_cluster\", instance_id=~\"$instance_id\"}[$__range]))",
"hide": false,
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "",
"type": "stat"
}
Prometheus may return inaccurate results from increase() function because of the chosen data model - see this issue for details.
If you need accurate results, then the following options exist:
- To use
offset. Try something like the following:sum(http_server_requests_seconds_count - http_server_requests_seconds_count offset $__range). Note that this approach works only if the given metric -http_server_requests_seconds_countwasn't reset to 0 (akacounter reset) on the given time range. - To use
increase()function from MetricsQL. It returns accurate values - see these docs for details.
Hi everyone!
I'm kind of a newbie when it comes to Grafana, so I may have overlooked something easy.
I have been asked to make a dashboard displaying a large number of counters (>500). The values of the counters come from sensors that are physically ordered in a certain way (let's say Counter0 to Counter500 from left to right), so it's important to keep them displayed in the same order.
While the user would like to be able to know the precise value of each counter, what matters the most to them is to have a quick overview of the distribution of those values, for example to immediately see that counters on the left have significantly lower values than counters on the right.
My first instinct was to use a Bar Gauge panel but it becomes a mess when there are too many values to display.
User would be happy with the ability to "zoom in and out" of the panel, for example to have a quick overview to see whether the values are more or less equally distributed and then "zoom in" to see which precise counter is different, but I don't see how to do that with Grafana.
Does anyone have any suggestion?
Hi Prometheus community,
I’m relatively new to Prometheus, having previously used InfluxDB for metrics. I’m struggling to visualize a simple counter (http_requests_total) in Grafana, and I need some advice. Here’s what I’m trying to achieve:
-
Count graph, NOT rate or percentage: I want the graph to show the number of requests over time. For example, if I select “Last 6 hours,” I want to see how many requests occurred during that time window.
-
Relative values only: I don’t care about the absolute counter value (e.g., "150,000" at some point). Instead, I want the graph to start at 0 for the beginning of the selected time window and show relative increments from there.
-
Smooth increments: I don’t want to see sharp peaks every time the counter increments, like what happens with
increase(). -
Adaptable to any time frame: The visualization should automatically adjust for any selected time range in Grafana.
Here’s an example of what I had with InfluxDB (attached image). It shows the actual peaks and their sizes in absolute numbers over time, which is exactly what I need.
I can’t seem to replicate this with Prometheus. Am I missing something fundamental?
Thanks for your help!
Hello everyone !
I have a service which expose a counter. That counter is inc of 1 every 10s for example. I would like to display that total value in grafana like this, with increase function. Grafana says that increase function manage pod restart.
Problem came when my service restart for any reason, my counter go back to 0. But i would like in grafana that my new counter start to the last value (lets say here 22) and not from 0.
First screenshot use increase with $__range of 3hours, which seem to working nicely. But when i change timerange from 3h to 1h for example, when i have a restart i have that dashboard
I don't have my linear function that i would, i don't know why my curve is straight and do not increase. If i take more range, sometime that work, sometime i got decrease, which should never happen with a counter...
Thanks for your help :)
That's a correct way to do it. You can also use increase() which is syntactic sugar for using rate() that way.
Can someone explain how the range selector
This is only used by Prometheus, and indicates what data to work over.
the Step and Resolution settings in grafana influence each other?
This is used on the Grafana side, it affects how many time slices it'll request from Prometheus.
These settings do not directly influence each other. However the resolution should work out to be smaller than the range, or you'll be undersampling and miss information.
The 3600 * sum(rate(signup_total[1h])) can be substituted with sum(increase(signup_total[1h])) . The increase(counter[d]) function returns counter increase on the given lookbehind window d. E.g. increase(signup_total[1h]) returns the number of signups during the last hour.
Note that the returned value from increase(signup_total[1h]) may be fractional even if signup_total contains only integer values. This is because of extrapolation - see this issue for technical details. There are the following solutions for this issue:
- To use offset modifier:
signup_total - (signup_total offset 1h). This query returns correct results ifsignup_totalwasn't reset to zero during the last hour. In this case thesum(signup_total - (signup_total offset 1h))is roughly equivalent tosum(increase(signup_total[1h])), but returns more accurate integer results. - To use VictoriaMetrics. It returns the expected integer results from
increase()out of the box. See this article and this comment for technical details.