Let's say that I have to monitor CPU utilization and create alert lifestyle based upon some thersold values , or reading some DB to get live time metrices of an app and create alerts .

junior question so, have mercy:

I'm using grafana mostly to monitor. but as its a small app with not a lot of users, not much worry. but we did have some trouble with overloading cpu-probably due to bad coding in core.

so question is for example, my boss wanted me to export pdf's and mail them to myself of dashboards of grafana- which isn't possible in OSS version. (reports available only in license status)

so i looked into prometheus expression browser thinking to export from there. got some progress.

but looking at kubectl top command. why wouldn't i simply put a script to alert me everytime the node reaches lets say 90% cpu?

with same on memory usage?

why should i use the granulated, and although lovely and detailed, version of grafana, if i can simply get it via alerts- as in, simple and effecient. why would i need the granular resolution of grafana/ prometheus?

I can do a simple awk command from kubectl top, to alert me.. using a job.

Exploring prometheus for a while...
Wondering if there is a visualization plugin so that I remove grafana integration.
Did anyone figure out or is already using charts and visualization inside prometheus? If yes what would you recommend?

Both Prometheus and Grafana have alerting mechanisms. From the point of view of best alerting practices, how do you decide whether to create your alerts in Prometheus or in Grafana when both are installed on your data center?

I am currently in the process of setting up data visualization using Grafana, but I am unsure whether to use InfluxDB or Prometheus as my database. From my understanding, data needs to be pushed to InfluxDB, would that require a push service, such as a small Python server, to pull data from sources that do not support pushing data and then push it to InfluxDB when new data is present? (It doesn't bother me to code a small server, it's more a question of whether I understand the service structure correctly?)

My use case involves obtaining data from various sources, including Proxmox, TrueNAS, and other services in my home lab, as well as IoT devices on my local network which I have control over, so I guess it’s not a problem, but I also need to get data from external API's, some of which do not support data pushing.

I find it quite difficult to choose what is right in my case…

Hi all,

Recently I picked up a project in my company to redefine our observability domain. On the topic of alerting, we previously were using a mix of Grafana alerts with Prometheus alerts. It is messy and all over the place to have alerts defined in both places.

Now I want to unify everything under one solution so I took a good look at both software and here are my findings so far:

Prometheus AlertManager:

Pros

• Very robust and battle-tested

• Possible to have it fully automated

• Available as part of Managed Prometheus offering by GCP (which we are hosted on)

• Supports automation as GKE custom resources so it can be integrated into our GitOps suite

Cons

• Not very user-friendly

• Unable to link it to Grafana Dashboards

Grafana AlertManager:

Pros

• User friendly

• Possibility to visualize using GUI

• Able to link to dashboards so it is much easier to investigate the issue

Cons

• Not great in terms of automation

• I mean you either have to use Terraform or Grizzly none of which fits well with our GitOps config

So if unclear, I was mostly inclined to go with Grafana alerting but the automation part is very important for me. If I can't find a good solution for automating Grafana alerts I'll go with Prometheus alerting.

Is there any part of the picture that I'm missing here? Any better solution than these two you can suggest?

Thank you

Hi folks — we’re evaluating whether it’s worth switching from standalone Prometheus to Grafana Mimir, mainly for performance and efficiency gains.

Our current setup is two independent Prometheus servers collecting metrics, with Promxy providing a unified query layer.

If you have experience with this, or know of any solid blog posts / benchmarks that compare them, we’d really appreciate pointers — especially around:

• Query performance: How does Mimir (HA + MinIO backend) perform for long-range queries (6+ months) compared to querying local Prometheus TSDB?

• Storage efficiency: How does Mimir’s storage usage typically compare to local Prometheus storage for the same retention?

• Quorum / minimum footprint: Does Mimir require at least 3 hosts (or similar) for quorum/high availability, and what’s the practical minimum deployment size for HA?

Thanks in advance!

I've got a few small servers I want to take metrics from, ideally system resources AND logs. Do I need both prometheus and loki? Or prometheus and influx?

Any solution for OPNSense?

When setting up alerting for your infrastructure, do you find it better to use Alertmanager or the native alerting in grafana?

In an environment where you have Prometheus and Grafana installed, what are the factors that would lead you to decide whether to create your alerts in one versus the other? Are there some best practices around this?

Happy to share my analysis of Prometheus Vs Grafana Vs graphite, which one will last? https://www.loomsystems.com/blog/single-post/2017/06/07/prometheus-vs-grafana-vs-graphite-a-feature-comparison

Hi there, I'm sure this question has been asked several times, or perhaps my Google-fu is off today. But I am struggling to understand which of these to use, for my specific use case.

I found these blog posts which sum up most things pretty well:

• https://prometheus.io/blog/2021/11/16/agent/

• https://grafana.com/blog/2021/11/16/why-we-created-a-prometheus-agent-mode-from-the-grafana-agent/

Recently our team just set up Loki as our centralized logging system for all of our docker swarm nodes and it works great, using fluentd as the log scraper/shipper, and of course Grafana for the frontend of all of this.

Now, we are focusing on metrics, and I am trying to set up Prometheus, but I have read that this project is very opinionated in the fact that it is primarily designed with PULL configuration in mind. Now I understand why this is and I feel that the docs do a solid job explaining this preference, but also happy to see there has been more recent support for Prometheus in PUSH configurations. For our small dev team, we want something similar to how Loki is set up, so we ideally we'd want just a single Prometheus server, with an agent (or set of agents) deployed onto whatever hosts we want to monitor. The reasons for this come down to our current tech stack, our CI/CD tooling and how often we rotate/refresh hosts.. I don't want to have to add/configure "multiple" prometheus instances into Grafana, every time we set up a new cluster/host, and keeping track of all those URLs etc.. I just see the PUSH configuration being so much easier to manage. If my thinking on this is completely wrong, please feel free to enlighten me :)

Now, I have been reading a ton of documentation over the last few days and even found some articles (linked above) explaining how Prometheus "Agent" mode is actually newer than grafana-agent, but basically uses the already-proven code as its upstream. It makes total sense to pull that into the main Prometheus project, but as a user, I am now confused more than ever on which one I should be using for the best support and which one will eliminates any future tech debt.

Here are some questions:

• Based on my recent reading of the docs, the Grafana-Agent has "embedded" versions of popular agents like cadvisor and node_exporter already built-in, which means I would ONLY have to deploy the single Grafana-Agent onto each of my hosts, correct? Traditionally these "embedded" agents would each be deployed on every host and you would have Prometheus scrape from them right?

• I understand that Prometheus in "Agent" mode has several features disabled which makes it focus more on the "remote_write" PUSH feature. But other than that, is it basically the same thing as Grafana-Agent, except with the absence of the built-in exporter agents (e.g. cadvisor, node_exporter, etc..)?

• Is one significantly more mature/robust than the other in it's current state? To my knowledge the Grafana-Agent code came first and is essentially the upstream source for Prometheus "Agent" mode. Making the Grafana-Agent the recommended choice for production right now? Which one would you reccommend, and why?

• Since Prometheus in "Agent" mode is newer and pulled into the official Prometheus project, do you think that the Grafana-Agent project will eventually lose popularity and become unmaintained in the long-term future? I'm afraid that If I go with Grafana-Agent now, I might have to switch to using Prometheus "Agent" mode later on. Not that it'd be too difficult to switch, lol.

• Any other thoughts or recommendations to a PUSH configuration approach for Prometheus metrics?

Sorry for the massive wall of text and I hope this is the right sub for these questions. If not could someone kindly point me in the right direction?

Thank you so much!

Hi everyone,

at the moment we're running Zabbix with hosts like Proxys, Linux servers and Docker containers. But I don't know: Zabbix feels a little bit "unmodern". I took a look on Grafana (with Zabbix as data source) and it's made awesome. I think, I don't wanna run it like that permanently 'cause sometimes getting data from the Zabbix data source is taking too long.

Now I'm thinking about switching completely to Grafana. I just want a real time monitoring system for our infrastructure which feels really handsome.

Some questions:
- in general: what do you use?
- Prometheus as a data source or Zabbix? How does Prometheus info getting works? Zabbix you got an agent but Prometheus? Is it faster?
- I would run Prometheus / Grafana in Docker containers. Do you recommend it?
- Docker monitoring possible with Prometheus / Grafana?

Thank you so much for your help. Have a nice day. :)

A systems administrator job that I found for a company I’d love to work for was recently posted and they mentioned monitoring tools, Prometheus and Grafana experience specifically as a plus. Does anyone have any good homelab recommendations for using these tools and getting a better understanding how they work?

In one of my environments I have libreNMS pulling from about 20-30 devices, at a frequency of once per minute. Almost all of the endpoints are SNMP and in a few cases IPMI.

Each minute when the pass happens all CPU cores on the libreNMS system get pegged for tens of seconds, and each endpoint gets a tangible CPU increase too. Underlying storage is bottlenecked for performance but I'm not so sure that's the source of the usage here, but mentioning it as it might be relevant. Anyways... I've just accepted this as the cost of doing this.

But when I look at a few examples of Prometheus+Grafana+related around me (friend that set their own up, and one set up at a client site I am responsible for) it looks like it gives me the same, or better, fidelity but at way lower resource (CPU?) usage to "do the same" (unsure about IPMI).

Is it just me or is Prometheus+Grafana+related ludicrously more efficient than libreNMS or is my environment abnormal in results?

edit: I KNOW THEY ARE DIFFERENT TECHNOLOGIES. STOP TELLING ME THAT.

I've recently set up Prometheus and Grafana for monitoring, and I've configured a few basic alerts so far—specifically for disk space, memory, and CPU usage. While these are essential, I feel there’s a lot more I can do with Prometheus and Grafana to keep our systems running smoothly and efficiently.

What I'm looking for:

• Recommended Metrics for Alerts: What are some other critical metrics you monitor? Are there specific API-related metrics, network metrics, or application-level metrics that you find especially valuable?

• Best Practices: How do you determine the thresholds for your alerts? Any tips on avoiding alert fatigue while still catching important issues early?

• Advanced Use Cases: Are there any advanced or creative ways you're using Prometheus and Grafana that have significantly improved your monitoring and alerting? For instance, alerts based on latency, error rates, or business-specific metrics.

• Visualization Tips: Any suggestions for Grafana dashboards that provide clear and actionable insights? I'm interested in seeing examples of how you visualize complex data effectively.

I appreciate any insights or suggestions you have! Whether you’re monitoring a small app or a large infrastructure, I’d love to hear about your experiences and what’s worked well for you.

Thanks in advance

Hello, I am wondering whether there are open source alternatives to Grafana when it comes to displaying metrics from Prometheus (or any other TSDB)? It feels like Grafana is the de-facto standard. I have become quite frustrated by the experience that Grafana offers: it is slow to render, editing UI is bloated and confusing, it is not collaborative (e.g., real-time modificaitons or comments), it is heavy and unflexible (e.g., I would like to be able to create lightweight copies of dashboards during incidents).

Do others feel the same? Do you have alternatives to propose (preferably open source)?

Hello, I have good programming skills but i have never tried or built something that requires logging and monitoring. I am new to this. I have to create a dashboard for a platform. It has 2 main components Nginx and backend in Nodejs. They generate log files everyday. I want to built a dashboard so that i can monitor my vm on which the platform is running and logs which are generated. I will have a main machine where grafana and all other tools will be installed but i can have many vms which will have same platform running. Please help me how can i do so. And how can i make something that is easily installable on other vms i create in future running same thing.

Olá, pessoal! Tudo bem?

Estou procurando maneiras eficazes de monitorar minhas aplicações e gostaria de saber a opinião de vocês sobre a utilização do Grafana e Prometheus. Estou procurando soluções free, pois meu objetivo é reduzir custos. No entanto, nunca usei essas ferramentas antes e estou um pouco perdido sobre como começar.

Gostaria de ouvir dicas e sugestões sobre como implementar essas ferramentas da melhor forma possível. Quais são as melhores práticas? Existem armadilhas comuns que devo evitar? Qual é a curva de aprendizado?