For the singlestat panel, you can just sum the two metrics and then add them together. Here is an example with two different metrics:
sum(prometheus_local_storage_memory_series) + sum(counters_logins)
Recommended reading, just in case you are doing anything with rates as well: https://www.robustperception.io/rate-then-sum-never-sum-then-rate/
Answer from Daniel Lee on Stack OverflowThis is the expected behavior when using a binary operator: both side must have a matching label set to be taken into account.
If you want to be able to aggregate both side and get the single one, you first must get the union of different metrics using the __name__ label:
sum by(__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"})
You can cascade the aggregation operator:
sum by (type) (sum by (__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"}))
Finally, you can also compact everything into:
sum by (type) ({__name__=~"metric_a|metric_b",job=~"provision-dev"})
The following PromQL query should sum metric_a and metric_b by type:
(sum(metric_a) by (type) + sum(metric_b) by (type))
or
(sum(metric_a) by (type) unless sum(metric_b) by (type))
or
(sum(metric_b) by (type) unless sum(metric_a) by (type))
How it works:
- The
sum(metric_a) by (type) + sum(metric_b) by (type)sums time series with matchingtypelabel values on both sides of+according to matching rules - The
sum(metric_a) by (type) unless sum(metric_b) by (type)returnssum(metric_a) by (type)results fortypelabel values missing insum(metric_b) by (type). See docs aboutunlessoperator. - The
sum(metric_b) by (type) unless sum(metric_a) by (type)returnssum(metric_a) by (type)results fortypelabel values missing insum(metric_a) by (type).
Then results from these three queries are joined with or operator.
This query is equivalent to the query proposed by Michael: sum({__name__=~"metric_a|metric_b"}) by (type) .
P.S. This query can be simplified further when using MetricsQL:
sum(metric_a, metric_b) by (type)
This query works, since sum() function in MetricsQL accepts and sums arbitrary number of arguments.
Sum two metrics from two different prometheus datasources in a single graph
Visualizing two metrics on the same graph
How to combine two metrics into one in Prometheus
Prometheus - How to “join” two metrics and calculate the difference?
Hello, i've been testing prometheus 1.4.1 and i've been wondering if it is possible to put two different metrics (for example memory used and memory free) on the same graph.
I've been looking for a way to do that but so far no luck; does this means that the best way to display metrics is not by using prometheus itself but attaching to grafana and using grafana capabilities ?
Also i saw robustperception post on how to check if ssh is responding or not ; is there a way - using prometheus - to see which process name is listening to a given port (this for sanity check purpose)?
edit: corrected one question
I am new to Prometheus metrics and I still struggle to see the right way to do things.
I recently discovered that the job I created had one target that was reading two instances metrics. That means that I had two sequences merged into one and the counter was going up and down.
After a question at Stackoverflow I changed the job to
- job_name: 'example'
scrape_interval: 30s
params:
-instance: ['372c43937e6acd56073246ffdf0e95597480cdba24688cc331d77a8c3c07e1a9','be007691832f4002a86cf0148768a3b039304e9b1fa7ae91a032b0beddc152b2']
static_configs:
- targets: ['example.com']
metrics_path: /metrics
scheme: httpsThat way I get two sequences of metrics within the same job name.
Is there anything that can group these two sequences into one?
I know that I can query the Prometheus with a sum and combine them in every query, but I would like to simplify this and query them as a single sequence.