I found here this solution: {__name__=~"metricA|metricB|metricC",container_name=~"frontend|backend|db"}.
I found here this solution: {__name__=~"metricA|metricB|metricC",container_name=~"frontend|backend|db"}.
You can use the or operator, however this does not generalise as it ignores metric names. I'd suggest making multiple queries to the API.
How to execute a query with two metrics in Prometheus? - Stack Overflow
Visualizing two metrics on the same graph
grafana - Query multiple metrics in one Prometheus HTTP Call - Stack Overflow
Multiple Prometheus queries
First of all, for more complex graphing you should definitely investigate Grafana. The built-in Prometheus graphs are useful eg. for debugging, but definitely more limited. In particular one graph will only display the results of one query.
Now for a hack that I definitely do not recommend:
flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"}
or
label_replace(flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}, "distinct", "foo", "job", ".*")
Since, as documented
vector1 or vector2results in a vector that contains all original elements (label sets + values) ofvector1and additionally all elements ofvector2which do not have matching label sets invector1.
you can add a new label that is not present in the labels on the first vector to the second vector and thus keep all elements from both.
It is possible to select multiple metric names with a single PromQL query by using a regular expression filter on __name__ label:
{__name__=~"flink_taskmanager_job_task_operator_numRecords(In|Out)PerSecond",operator_name="Map"}
See docs about the __name__ label here.
There is another solution when using Prometheus-compatible query engine such as MetricsQL by using union function:
union(
flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"},
flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}
)
Note that selecting multiple time series via __name__ regexp can result in vector cannot contain metrics with the same labelset error if the selected series are wrapped in any PromQL function. For example:
max_over_time(
{__name__=~"flink_taskmanager_job_task_operator_numRecords(In|Out)PerSecond",operator_name="Map"}[5m]
)
This is because Prometheus removes metric names from input series when applying PromQL functions. MetricsQL from VictoriaMetrics provides a solution for this issue - keep_metric_names modifier (see these docs for details):
max_over_time(
{__name__=~"flink_taskmanager_job_task_operator_numRecords(In|Out)PerSecond",operator_name="Map"}[5m]
)
keep_metric_names
P.S. I work on VictoriaMetrics and MetricsQL.
Hello, i've been testing prometheus 1.4.1 and i've been wondering if it is possible to put two different metrics (for example memory used and memory free) on the same graph.
I've been looking for a way to do that but so far no luck; does this means that the best way to display metrics is not by using prometheus itself but attaching to grafana and using grafana capabilities ?
Also i saw robustperception post on how to check if ssh is responding or not ; is there a way - using prometheus - to see which process name is listening to a given port (this for sanity check purpose)?
edit: corrected one question
You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):
(
max(consul_health_service_status{status="critical"})
by (service_name,status,node) == 1
)
+ on(service_name,node) group_left(env)
(
0 * consul_service_tags
)
The important part here is the operation + on(service_name,node) group_left(env):
- the
+is "abused" as a join operator (fine since0 * consul_service_tagsalways has the value 0) group_left(env)is the modifier that includes the extra labelenvfrom the right (consul_service_tags)
It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.
The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.
For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:
consul_health_service_status
* on(service_name, node) group_left(env)
consul_service_tags
Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:
consul_health_service_status{status="critical"}
* on(service_name, node) group_left(env)
consul_service_tags