prometheus_tsdb_head_series
Just because I always forget and have to Google this, and this question is at the top. As per this answer there will likely be a variance between prometheus_tsdb_head_series & count({__name__=~".+"}) because of variations in what they consider "active", but unless you have a large variability in metric count, I'd recommend prometheus_tsdb_head_series because if you have a lot of metrics, it's a lot faster to query.
Hi,
I'm trying to troubleshoot Prometheus high memory usage. While I can view my top x metrics with high cardinality, to get an idea, I want to know the total number of metrics that prometheus is currently storing in tsdb. Is there a query that can be useful here?
Thanks In advance.
How is it possible to know the total number of unique metrics in Prometheus - Stack Overflow
Number of metrics one Prometheus server can handle?
How do I show the count of a specific group of metrics from a prometheus histogram? - Stack Overflow
Prometheus: Count metric value over a period of time - Stack Overflow
prometheus_tsdb_head_series
Just because I always forget and have to Google this, and this question is at the top. As per this answer there will likely be a variance between prometheus_tsdb_head_series & count({__name__=~".+"}) because of variations in what they consider "active", but unless you have a large variability in metric count, I'd recommend prometheus_tsdb_head_series because if you have a lot of metrics, it's a lot faster to query.
How about count({__name__=~".+"})?
It does return number of time series in the database. I compared with amount of metrics currently exposed by each target by manual scrapping, and it matches it +/- 10%. I guess the difference is due to some targets that I had in the past and now they are offline.
rate(rpc_request_duration_seconds_bucket{le="+Inf"}[1m])
- ignoring(le)
rate(rpc_request_duration_seconds_bucket{le="1.0"}[1m])
will return how many queries are going over 1s every second.
This is all queries, minus the queries that take less than or equal to one second.
You want to write something like
rate(rpc_request_duration_seconds_count [1m])
This will give you the requests per second in a (sliding) interval of 1 minute. See the Query functions doc of Prometheus.
If the metric can have either 0 or 1 values, then the sum_over_time(metric[d]) calculates the number of 1 values on the specified lookbehind window d. For example, sum_over_time(up[1h]) returns the number of up samples with 1 value during the last hour. The number of 0 values then can be calculates as count_over_time(up[1h]) - sum_over_time(up[1h]).
If the metric can have other values than 0 and 1, then Prometheus doesn't provide functions for counting the number of samples with a particular value yet :(
There is another Prometheus-like system, which allows counting the number of raw samples with the given value on the specified lookback window - VictoriaMetrics (I'm the core developer of this system). It provides count_eq_over_time function for this task. For example, the following MetricsQL query returns the number of samples for some_metric time series with the value 42 over the last hour:
count_eq_over_time(some_metric[1h], 42)
It looks like you are using Grafana as a visualisation tool. There is the possibility to add an expression to a panel. This would result in a second Metric (in my example below B). This is located in the Query Tab on Edit. You ma need a newer Grafana version (AFAIK it was introduced with v6.x)
