Recording rules run the query you specify on a timer, and write the output of that query to the TSDB. So if you configure a rule like this: - record: job:up:avg expr: avg without (instance) (up) It will take metrics like this: up{job="node",instance="foo:9100"} 1.0 up{job="node",instance="bar:9100"} 1.0 up{job="web",instance="foo:8080"} 1.0 up{job="web",instance="bar:8080"} 0.0 And store this in the TSDB: job:up:avg{job="node"} 1.0 job:up:avg{job="web"} 0.5 It will run this at whatever interval you specify in the rule group. More on reddit.com

Modify Prometheus Configuration: Instead of modifying the ServiceMonitor directly, you should change the Prometheus configuration. This can be done by editing the Prometheus custom resource in the openshift-monitoring namespace. Adjust the scrape_interval setting to a higher value to reduce the frequency of scrapes. Use Retention and Storage Settings: Adjust the data retention period and storage size in the Prometheus configuration. This can help manage the resource consumption.Scale Up Prometheus Instances: Consider scaling up the number of Prometheus instances to distribute the load. Optimize or Reduce Targets: Analyze if all targets are necessary or if some metrics can be dropped. This reduces the amount of data Prometheus needs to scrape and store. Monitor and Adjust Resource Allocation: Continuously monitor the resource utilization of Prometheus and adjust CPU and memory requests and limits accordingly. Implement Thanos or Remote Storage: For long-term storage and efficient querying, consider integrating Thanos or another remote storage solution. More on reddit.com

For future travelers here is what I found after looking around.Prometheus cant do what I'm asking for directly since running the sum (aggergation) query requires the data to be in the tsdb , which deafets the purpose of doing the aggregation on scrap. There are other ways to do this. writing your own exporter; basically hack a python script and expose /metrics that scrap rabbitmq and do the aggregation on demand without having a state. having another prometheus instance with a recording rule federated to the main instance, this will be the most compatible but will cause delays of queries since recording rules gets evaluated after the scrap getting stored using victoriametrics aggregation via a cheap single instance server with the lowest retention possible then using the /federate endpoint as a scrap endpoint for my main prometheus give up and just dont collect those metrics I choose the 3rd option since it is the most sane one. Although I dont like having this victoriametrics instance just to aggregate, and I dont want to maintain or solve the problems related to metrics aggregation with my own hacky python script, I have no other choice but to use this. If anyone else have a better solution let me know edit: note with victoriametrics, the only shared metrics with the federate endpoint are the aggregated ones, so you might need to scrap twice, once for prometheus with dropping the labels you want to aggregate, and second for victoriametrics with the labels you want to aggregate More on reddit.com

Today faced same issue :(( More on reddit.com