This is the expected behavior when using a binary operator: both side must have a matching label set to be taken into account.
If you want to be able to aggregate both side and get the single one, you first must get the union of different metrics using the __name__ label:
sum by(__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"})
You can cascade the aggregation operator:
sum by (type) (sum by (__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"}))
Finally, you can also compact everything into:
sum by (type) ({__name__=~"metric_a|metric_b",job=~"provision-dev"})
Prometheus : how do i sum by with 2 different metrics - Stack Overflow
prometheus - Difference between PromQL "by" and "without" unclear - Stack Overflow
prometheus - PromQL sum by label over time - Stack Overflow
promql function to sum distinct for gauge value
What is PromQL?
How to use group by in PromQL?
How do I calculate a per-second rate in PromQL?
Hello,
I have this graph monitoring the bandwidth of a VLAN on a switch every 1m using SNMP Exporter, but I also what to get the total/sum data over time, so if I select the last hour it will show x amount inbound and x amount outbound.
sum by(ifName) (irate(ifHCInOctets{instance=~"192.168.200.10", job="snmp_exporter", ifName=~".*(1001).*"}[1m])) * 8My current graph:
I'd like to duplicate and create a stat panel show how much data in total has passed over what period I choose that's all.
For the metric I'm not sure whether to use bytes(SI) or bytes(IEC), but are similar if I change to either.
Not sure how to calculate this, but I have this created for the past 1 hour.
by copying the PromQL in Grafana and changing to a stat panel and then editing to use this:
Not sure if this is ok as I'm not sure how to calculate it all, maths was never my best subject.
Any help would be great.
I think something like is close: with sum_over_time
sum by(ifName) (sum_over_time(ifHCInOctets{instance=~"192.168.200.10", job="snmp_exporter", ifName=~".*(1001).*"}[1m])) * 8but it comes back as 85.8 Pib when it should be 85.8 TB with my calculations.
EDIT
Observium:
What Grafana shows
This is the expected behavior when using a binary operator: both side must have a matching label set to be taken into account.
If you want to be able to aggregate both side and get the single one, you first must get the union of different metrics using the __name__ label:
sum by(__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"})
You can cascade the aggregation operator:
sum by (type) (sum by (__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"}))
Finally, you can also compact everything into:
sum by (type) ({__name__=~"metric_a|metric_b",job=~"provision-dev"})
The following PromQL query should sum metric_a and metric_b by type:
(sum(metric_a) by (type) + sum(metric_b) by (type))
or
(sum(metric_a) by (type) unless sum(metric_b) by (type))
or
(sum(metric_b) by (type) unless sum(metric_a) by (type))
How it works:
- The
sum(metric_a) by (type) + sum(metric_b) by (type)sums time series with matchingtypelabel values on both sides of+according to matching rules - The
sum(metric_a) by (type) unless sum(metric_b) by (type)returnssum(metric_a) by (type)results fortypelabel values missing insum(metric_b) by (type). See docs aboutunlessoperator. - The
sum(metric_b) by (type) unless sum(metric_a) by (type)returnssum(metric_a) by (type)results fortypelabel values missing insum(metric_a) by (type).
Then results from these three queries are joined with or operator.
This query is equivalent to the query proposed by Michael: sum({__name__=~"metric_a|metric_b"}) by (type) .
P.S. This query can be simplified further when using MetricsQL:
sum(metric_a, metric_b) by (type)
This query works, since sum() function in MetricsQL accepts and sums arbitrary number of arguments.
- The
bymodifier groups aggregate function results by labels enumerated insideby(...). - The
withoutmodifier groups aggregate function results by all the labels except those enumerated insidewithout(...).
For example, suppose process_resident_memory_bytes metric exists with job, instance and datacenter labels:
process_resident_memory_bytes{job="job1",instance="host1",datacenter="dc1"} N1
process_resident_memory_bytes{job="job1",instance="host2",datacenter="dc1"} N2
process_resident_memory_bytes{job="job1",instance="host1",datacenter="dc2"} N3
process_resident_memory_bytes{job="job2",instance="host1",datacenter="dc1"} N4
Then sum(process_resident_memory_bytes) by (datacenter) would return summary per-datacenter memory usage, while sum(process_resident_memory_bytes) without (instance) would return summary per-job per-datacenter memory usage.
All of these examples are aggregating incorrectly, as you're averaging an average. You want:
sum without (path,host) (
rate(request_duration_sum{status_code=~"2.*"}[5m])
)
/
sum without (path,host) (
rate(request_duration_count{status_code=~"2.*"}[5m])
)
Which will return the average latency per status_code plus any other remaining labels.
The following PromQL query should return the increase of bot_guides_failed_total time series over the last hour (see 1h in square brackets) grouped by searchTerm label:
sum(increase(bot_guides_failed_total[1h])) by (searchTerm)
If this query is used for building a graph in Grafana, then every point on the graph will contain the increase of the bot_guides_failed_total metric over the one hour interval ending at this point.
Note that the increase() function in Prometheus may return unexpected results when applied to slow-changing counters. This is because of extrapolation - see this issue for details. If you need the expected results from the increase() function, then try VictoriaMetrics instead - this is Prometheus-like monitoring system I work on.
You need to use a function to extract the number of searches in a specific time window. E.g:
sum by (increase(searchTerm[5m])) (bot_guides_failed_total)
This will give you the number of searchTerm grouped by bot_guides_failed_total in a 5 minutes time window.