prometheus sum by label

stackoverflow.com › questions › 45154993 › how-can-i-group-labels-in-a-prometheus-query

promql - How can I group labels in a Prometheus query? - Stack Overflow

sql - How to group on labels in aggregate query to Promscale

1 of 3

74

It's even easier

sum by (group) (my_metric)

2 of 3

57

Yes, you can you use label replace to group all the misc together:

sum by (new_group) (
  label_replace(
    label_replace(my_metric, "new_group", "$1", "group", ".+"),
    "new_group", "misc", "group", "misc group.+"
  )
)

The inner label_replace copies all values from group into new_group, the outer overwrites those which match "misc group.+" with "misc", and we then sum by the "new_group" label. The reason for using a new label is the series would no longer be unique if we just overwrote the "group" label, and the sum wouldn't work.

Prometheus group by substring of label

Prometheus: PromQL - Adding a label to the metric

Prometheus Query to aggregate metrics by certain label values

prometheus - How to add label value from different metric?

More results from stackoverflow.com

Prometheus

prometheus.io › docs › prometheus › latest › querying › examples

Query examples | Prometheus

If we have two different metrics with the same dimensional labels, we can apply binary operators to them and elements on both sides with the same label set will get matched and propagated to the output. For example, this expression returns the unused memory in MiB for every instance (on a fictional cluster scheduler exposing these metrics about the instances it runs): (instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024 · The same expression, but summed by application, could be written like this:

Discussions

prometheus - PromQL sum by label over time - Stack Overflow

I am trying to sum up a counter metric over time and grouped by the label applied. I've tried this sum by (searchTerm) (bot_guides_failed_total) but I want to have out in a table form like this: More on stackoverflow.com

stackoverflow.com

Prometheus query to only include sum of labels with other metric value over some value - Stack Overflow

If I use this query sum by (host_name) ... / sum by (host_name) (vmware_host_memory_max) it looks like I have way over provisioned because there are quite a few VMs that are off but are still included in the total of vmware_vm_memory_max for a given ESXi host. I'd like to have Prometheus ignore any ... More on stackoverflow.com

stackoverflow.com

openshift - Prometheus sum while filtering by label - Stack Overflow

In Prometheus I can provide a sum of pod cpu usage that is filtered by namespace: sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{namespace='my-namespace'}) by (pod) I... More on stackoverflow.com

stackoverflow.com

relabel and aggregate metrics

For future travelers here is what I found after looking around.Prometheus cant do what I'm asking for directly since running the sum (aggergation) query requires the data to be in the tsdb , which deafets the purpose of doing the aggregation on scrap. There are other ways to do this. writing your own exporter; basically hack a python script and expose /metrics that scrap rabbitmq and do the aggregation on demand without having a state. having another prometheus instance with a recording rule federated to the main instance, this will be the most compatible but will cause delays of queries since recording rules gets evaluated after the scrap getting stored using victoriametrics aggregation via a cheap single instance server with the lowest retention possible then using the /federate endpoint as a scrap endpoint for my main prometheus give up and just dont collect those metrics I choose the 3rd option since it is the most sane one. Although I dont like having this victoriametrics instance just to aggregate, and I dont want to maintain or solve the problems related to metrics aggregation with my own hacky python script, I have no other choice but to use this. If anyone else have a better solution let me know edit: note with victoriametrics, the only shared metrics with the federate endpoint are the aggregated ones, so you might need to scrap twice, once for prometheus with dropping the labels you want to aggregate, and second for victoriametrics with the labels you want to aggregate More on reddit.com

r/PrometheusMonitoring

4

2

July 21, 2023

People also ask

What are the limitations of Prometheus labels?

Labels with high cardinality, like user_id or request_id—can generate too many time series, slowing down queries and increasing memory usage. Labels must follow specific naming rules, can't start with __, and are case-sensitive.

last9.io

Prometheus Group By Label: Advanced Aggregation Techniques for ...

What is the job label in Prometheus?

The job label identifies the target being scraped. It’s automatically assigned by Prometheus and helps group metrics by application, service, or source as defined in the scrape configuration.

last9.io

Prometheus Group By Label: Advanced Aggregation Techniques for ...

How can I group labels in a Prometheus query?

Use by (label1, label2) to retain specific labels or without (label) to exclude one or more. For example: sum(cpu_usage) by (region) sum(memory_usage) without (instance)

last9.io

Prometheus Group By Label: Advanced Aggregation Techniques for ...

Last9

Prometheus Group By Label: Advanced Aggregation Techniques for Monitoring | Last9

June 12, 2026 - In Prometheus, group by is an operator ... how data should be grouped. On the other hand, sum by (...) combines the sum() aggregation function with group by....

Better Stack

betterstack.com › community › questions › how-can-group-labels-in-prometheus-query

How Can I Group Labels in a Prometheus Query? | Better Stack Community

August 5, 2025 - If you want to count the total ... POST), you can use: ... This query sums the total requests and groups the result by the method label, giving you the total number of requests for each HTTP method....

Prometheus

prometheus.io › docs › prometheus › latest › querying › functions

Query functions | Prometheus

histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[10m]))) To aggregate all classic histograms, specify only the le label:

Prometheus

prometheus.io › docs › prometheus › latest › querying › operators

Operators | Prometheus

without removes the listed labels ... their label values are identical between all elements of the vector. sum(v) sums up sample values in v in the same way as the + binary operator does between two values....

Coralogix

coralogix.com › home › promql tutorial: 5 tricks to become a prometheus god

PromQL Tutorial: 5 Tricks to Become a Prometheus God

June 3, 2025 - The by and without clauses enable ... tells the query to include labels: the query sum by(instance) (node_filesystem_size_bytes) returns the total node_filesystem_size_bytes for each instance....

Find elsewhere

Google Bing Mojeek

Fiberplane

fiberplane.com › blog › why-are-prometheus-queries-hard

Why are Prometheus queries hard? | Fiberplane Blog

July 4, 2023 - When querying our data, we often care about some of those label dimensions and not others. In PromQL, sum by (label1, label2,…) groups the time series by the labels you specify and uses the sum function to merge series together (similar to ...

Medium

medium.com › @suchitasharma1106 › a-comprehensive-guide-to-grouping-and-functions-in-promql-cc3c438be320

A Comprehensive Guide to Grouping and Functions in PromQL | by Suchita Sharma | Medium

October 7, 2024 - Here are some common functions and how they can be applied: Aggregation functions are used to aggregate data across multiple time-series, typically with by() or without(). sum(): Adds up the values of all selected time series.

Promlabs

promlabs.com › promql-cheat-sheet

PromLabs | PromQL Cheat Sheet

Available aggregation operators: sum(), min(), max(), avg(), stddev(), stdvar(), count(), count_values(), group(), bottomk(), topk(), quantile()

Yannick Pereira-Reis

ypereirareis.github.io › blog › 2020 › 02 › 21 › how-to-join-prometheus-metrics-by-label-with-promql

How to join Prometheus metrics by label with PromQL – Yannick Pereira-Reis

sum(node_disk_bytes_read * on(instance) group_left(node_name) node_meta{}) by (node_name) on(instance) => this is how to JOIN on label instance. group_left(node_name) node_meta{} => means, keep the label node_name from metric node_meta in the result.

Robust Perception

robustperception.io › using-the-group-aggregator-in-promql

Using the group() aggregator in PromQL – Robust Perception | Prometheus Monitoring Experts

August 10, 2020 - That is first you aggregate away any other labels that don't matter with the inner aggregation (here just the mode label), and then count the resulting series with the outer aggregation. The thing is though that the inner aggregation doesn't have to be count. It could be sum, max, or even stddev, because it's not the numeric result of the aggregation that we care about but instead the labels on the resultant time series.

stackoverflow.com › questions › 71946587 › promql-sum-by-label-over-time

prometheus - PromQL sum by label over time - Stack Overflow

1 of 2

1

The following PromQL query should return the increase of bot_guides_failed_total time series over the last hour (see 1h in square brackets) grouped by searchTerm label:

sum(increase(bot_guides_failed_total[1h])) by (searchTerm)

If this query is used for building a graph in Grafana, then every point on the graph will contain the increase of the bot_guides_failed_total metric over the one hour interval ending at this point.

Note that the increase() function in Prometheus may return unexpected results when applied to slow-changing counters. This is because of extrapolation - see this issue for details. If you need the expected results from the increase() function, then try VictoriaMetrics instead - this is Prometheus-like monitoring system I work on.

2 of 2

0

You need to use a function to extract the number of searches in a specific time window. E.g:

sum  by  (increase(searchTerm[5m]))  (bot_guides_failed_total)

This will give you the number of searchTerm grouped by bot_guides_failed_total in a 5 minutes time window.

stackoverflow.com › questions › 70088980 › prometheus-query-to-only-include-sum-of-labels-with-other-metric-value-over-some

Prometheus query to only include sum of labels with other metric value over some value - Stack Overflow

sum by (host_name) (vmware_vm_memory_max and (vmware_vm_power_state == 1)) / sum by (host_name) (vmware_host_memory_max) * 100

SigNoz

signoz.io › guides › how to group labels in prometheus queries - a practical guide

How to Group Labels in Prometheus Queries - A Practical Guide | SigNoz

July 24, 2024 - Proper label grouping significantly improves query performance and data visualization. group() drops specified labels without modifying values, while sum() aggregates values across the grouped labels.

stackoverflow.com › questions › 77723318 › prometheus-sum-while-filtering-by-label

openshift - Prometheus sum while filtering by label - Stack Overflow

reddit.com › r/prometheusmonitoring › relabel and aggregate metrics

1 of 1

3

To filter the pod CPU usage further based on a specific pod label like application=my-app, you need to use the kube_pod_labels metric to filter by the desired label. You can try like this if it works:

sum(
  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{namespace='my-namespace'}
  * on(pod) group_left(label_application)
  kube_pod_labels{label_application='my-app'}
) by (pod)

You can read more from here

r/PrometheusMonitoring on Reddit: relabel and aggregate metrics

July 21, 2023 -

Hi,

I have rabbitmq metrics which contains the `channel` label. Since this label has high cardinality I decided I want to drop it, but faced an issue.

When prometheus drops it, there will be duplicates, and prometheus just take one of them, the exact situation here https://grafana.com/blog/2022/10/20/how-to-manage-high-cardinality-metrics-in-prometheus-and-kubernetes/#3-begin-optimizing-metrics in the `Reduce labels` section.

From what I can see, I need recording rule that would sum these metrics but im not sure about the order of operations.

If I have a metric_relabeling_rule in the scrapping config and a recording rule, which one will be applied first?

Is there a sensible way of recalculating all of the metrics that contains the `channel` label and take the sum of them such that no data is being dropped?

Or do I have to create a new metric name with the channel summed?

Edit:
In this response they say "maybe you need to aggregate over the duplicate series", but i dont know if they mean recording rules or what