I am currently in the process of setting up data visualization using Grafana, but I am unsure whether to use InfluxDB or Prometheus as my database. From my understanding, data needs to be pushed to InfluxDB, would that require a push service, such as a small Python server, to pull data from sources that do not support pushing data and then push it to InfluxDB when new data is present? (It doesn't bother me to code a small server, it's more a question of whether I understand the service structure correctly?)
My use case involves obtaining data from various sources, including Proxmox, TrueNAS, and other services in my home lab, as well as IoT devices on my local network which I have control over, so I guess it’s not a problem, but I also need to get data from external API's, some of which do not support data pushing.
I find it quite difficult to choose what is right in my case…
Hi all
We are currently investigating to replace our very expensive DataDog monitoring (That we are not completely satisfied with).
We are thinking to either use the full TICK stack from InfluxData (the OSS one) or Prometheus/AlertManager.
Which one would you recommend ? We will both monitor the infrastructure (On Google Cloud, with various services like SQL Server, PostgreSQL, RabbitMQ, MongoDB, NGINX and more), some application metrics that we will provide and also monitoring Kubernetes.
So far it looks like that Prometheus is more "hype" currently. I've been playing a little bit with both so far and for now I would more to toward the TICK stack that I think it more mature and flexible ? The only thing is that I could not find mature autodiscovery in Telegraf on Kubernetes (to monitor all the pods custom metrics automatically for example)
What would you recommend ? What would be the big differences between both. AFAIK both would both to PULL and PUSH (Prometheus being more PULL first)
For those who ask for the issues we have with DataDog:
Too expensive
Lot of weird limitations (can't monitor more than 100 RabbitMQ queues, can't monitor all the objects of a PostgreSQL database without listing all of them in their config files)
No support for autoscaling in Google Cloud where we asked as high priority almost 2 years ago, this will be a hard requirement in the next month's
Limited dashboards interoperability or exportability
Paid custom metrics
No configuration change history / rollback
Thanks
Hi, as a VMs monitoring system we have been using Sensu+InfluxDB for years (on-prem, multiple sites, > 500 VMs, VMWare). This system scale/works very well and also can be fully integrated with configuration management tool like Puppet, through which we can dynamically manage configurations, per-host parameters used by probes (e.g. credentials, probe parameters, etc.), per-host attributes (e.g. host tags) and also the discovery of services/hosts is fully automated. In addition to that, we are using Prometheus to monitor k8s and related services.
At the same time, the fate of Sensu and InfluxDB seems uncertain and subject to several changes, in addition to the fact that many services now come out natively with a Prometheus endpoint and a set native Grafana dashboards, so creating home-made dashboards and probes seems like a waste of time in 98% of cases.
-
In your opinion, should we change from Sensu to Prometheus in order to unify/standardize the monitoring system being used? Would you suggest any other tool?
-
If we decide to use Prometheus for VMs, is it worth thinking about using Consul for host discovery or is it a too complex solution? What would you use instead?
-
Regards timeseries DB, do you think is it better to migrate to another timeseries DB (e.g. Victoriametrics, M3DB) or not?
-
Based on your Prometheus experience, could Thanos (or similar sw) be a good solution (i.e. for aggregation/long term metrics store) or is it better to rely on a remote write to a dedicated timeseries DB?
I've got a few small servers I want to take metrics from, ideally system resources AND logs. Do I need both prometheus and loki? Or prometheus and influx?
Any solution for OPNSense?
Hi,
For my alerting and monitoring needs I generally tend to use grafana and influx. I recently came across prometheus and had few questions about it from the users who are using it.
I used telegraf to push custom metrics to influxdb. It is basically a push model. When I was going through prometheus architecture, it seems to be a pull model. Please help me understand how does it work
Let say I have installed prometheus. I have few machines behind scalesets which come and go depending on the load. How does prometheus know that there are new machines behind the scaleset which it needs to scrape. Also, Isn't a push better model than a pull one ?
Do we need to add new server connections in prometheus for it to scrape metrics from new server added?
We would like to have the capability to add business event driven monitoring and alerting to our solution at some point in future though the current use case is infrastructure and application monitoring.Mostly a java shop with some java script apps.We had issues with Influx which is our backend with Grafana currently(we are investigating adding Kapacitor and chronograph to the mix fairly soon) losing tons of data because of lack of clustering.
I'm trying to setup monitoring for my servers. I have one home server (repurposed hackintosh, so not really server hardware, just a regular PC running Linux) running several services mostly in Docker containers but a couple in VMs, I also have a couple VPS here and there. I have a PiHole running on a RasPi, libreelec/kodi running on another Pi, and opnsense running on dedicated hardware.
I'd like to monitor all these things to avoid stupid things like running out of space without noticing, or having a service down and not noticing, or being alerted if my borg backups fail, or warn me if my rsync.net account is approaching its quota, etc.
I have tried ELK about 6 months ago, but it didn't seem to do what I wanted: I could parse logs for metrics but alerting was a paid feature (a few thousand $ a year IIRC) so that was a no go.
I'm now giving Prometheus a shot because it does alerting for free. I have managed to put together an Ansible role to setup node-exporter wherever I want and put it behind a reverse proxy that does SSL and basic auth.
So it looks like node exporter is to Prometheus what Logstash is to ELK. Prometheus itself is a poor man's datastore filling the role of Ealsticsearch in ELK, but InfluxDB is better at it and recommended for keeping data longer term. Grafana is Kibana. Did I get that right?
-
Are ELK and Prometheus InfluxDB Grafana (PIG I guess?) redundant to each other or complementary?
-
Is InfluxDB mandatory?
-
What are Telegraf and fluentd for, aren't they doing the same thing as node-exporter?
-
Is there a guide to monitoring and alerting somewhere that would shed some light for someone who is new to it? I found blog posts and the projects' docs, but they assume a lot of prior knowledge on the topic that I don't have.
Hopefully these questions make sense, apologies if they don't; I'm very confused at it all.
Hello,
I am currently working on enhancing my code by incorporating metrics. The primary objective of these metrics is to track timestamps corresponding to specific events, such as registering each keypress and measuring the duration of the key press.
The code will continuously dispatch metrics; however, the time intervals between these metrics will not be consistent. Upon researching the Prometheus client, as well as the OpenTelemetry metrics exporter, I have learned that these tools will transmit metrics persistently, even when there is no change in the metric value. For instance, if I send a metric like press.length=6
, the client will continue to transmit this metric until I modify it to a different value. This behavior is not ideal for my purposes, as I prefer distinct data points on the graph rather than a continuous line.
I have a couple of questions:
-
In my use case, is it logically sound to opt for Prometheus, or would it be more suitable to consider another database such as InfluxDB?
-
Is it feasible to transmit metrics manually using StatsD
and Otel Collector
to avoid the issue of "duplicate" metrics and ensure precision between actual metric events?
I have data spread out over 3 systems: 2 TSDBs (prometheus and InfluxDBv2) and one MariaDB.
I was considering moving all of this into PostgreSQL+TimeScaleDB to have one back-end to manage. It'd make matching between timeseries and more static information stored in classic db tables easier.
-
anyone experience moving from either prometheus/influxdbv2 to timescale?
-
prometheus has this concept of exporters, agents which collect data from sources and make it available for scraping by prometheus. Is there something I could replace these with which would write data to TimeScaleDB?
I’m looking for a system monitor, and see many recommendations in previous threads. What I don’t understand is why there are so many combinations and what the differences are:
Prometheus + Grafana
Prometheus + Grafana + Netdata
InfluxDB + Telegraf + Grafana
Prometheus collects data. Grafana displays it. When would you need to add Netdata to that? Why would you drop Prometheus and go with both InfluxDB and Telegraf?
I need basic CPU, Memory, Storage, and Docker container monitoring for my main server. Would also like to monitor a remote VPS and local Linux clients with a single setup if possible.
What should I learn?
InfluxDB CEO and developer here. The next version of InfluxDB (0.9.5) will have our new storage engine. With that engine we'll be able to efficiently store either single event data or regularly sampled series. i.e. Irregular and regular time series.
InfluxDB supports int64, float64, bool, and string data types using different compression schemes for each one. Prometheus only supports float64.
For compression, the 0.9.5 version will have compression competitive with Prometheus. For some cases we'll see better results since we vary the compression on timestamps based on what we see. Best case scenario is a regular series sampled at exact intervals. In those by default we can compress 1k points timestamps as an 8 byte starting time, a delta (zig-zag encoded) and a count (also zig-zag encoded).
Depending on the shape of the data we've seen < 2.5 bytes per point on average after compactions.
YMMV based on your timestamps, the data type, and the shape of the data. Random floats with nanosecond scale timestamps with large variable deltas would be the worst, for instance.
The variable precision in timestamps is another feature that InfluxDB has. It can represent second, millisecond, microsecond, or nanosecond scale times. Prometheus is fixed at milliseconds.
Another difference is that writes to InfluxDB are durable after a success response is sent to the client. Prometheus buffers writes in memory and by default flushes them every 5 minutes, which opens a window of potential data loss.
Our hope is that once 0.9.5 of InfluxDB is released, it will be a good choice for Prometheus users to use as long term metrics storage (in conjunction with Prometheus). I'm pretty sure that support is already in Prometheus, but until the 0.9.5 release drops it might be a bit rocky. Obviously we'll have to work together and do a bunch of testing, but that's what I'm hoping for.
For single server metrics ingest, I would expect Prometheus to have better performance (although we've done no testing here and have no numbers) because of their more constrained data model and because they don't append writes to disk before writing out the index.
The query language between the two are very different. I'm not sure what they support that we don't yet or visa versa so you'd need to dig into the docs on both to see if there's something one can do that you need. Longer term our goal is to have InfluxDB's query functionality be a superset of Graphite, RRD, Prometheus and other time series solutions. I say superset because we want to cover those in addition to more analytic functions later on. It'll obviously take us time to get there.
Finally, a longer term goal for InfluxDB is to support high availability and horizontal scalability through clustering. The current clustering implementation isn't feature complete yet and is only in alpha. However, we're working on it and it's a core design goal for the project. Our clustering design is that data is eventually consistent.
To my knowledge, Prometheus' approach is to use double writes for HA (so there's no eventual consistency guarantee) and to use federation for horizontal scalability. I'm not sure how querying across federated servers would work.
Within an InfluxDB cluster, you can query across the server boundaries without copying all the data over the network. That's because each query is decomposed into a sort of MapReduce job that gets run on the fly.
There's probably more, but that's what I can think of at the moment.
We've got the marketing message from the two companies in the other answers. Now let's ignore it and get back to the sad real world of time-data series.
Some History
InfluxDB and prometheus were made to replace old tools from the past era (RRDtool, graphite).
InfluxDB is a time series database. Prometheus is a sort-of metrics collection and alerting tool, with a storage engine written just for that. (I'm actually not sure you could [or should] reuse the storage engine for something else)
Limitations
Sadly, writing a database is a very complex undertaking. The only way both these tools manage to ship something is by dropping all the hard features relating to high-availability and clustering.
To put it bluntly, it's a single application running only a single node.
Prometheus has no goal to support clustering and replication whatsoever. The official way to support failover is to "run 2 nodes and send data to both of them". Ouch. (Note that it's seriously the ONLY existing way possible, it's written countless times in the official documentation).
InfluxDB has been talking about clustering for years... until it was officially abandoned in March. Clustering ain't on the table anymore for InfluxDB. Just forget it. When it will be done (supposing it ever is) it will only be available in the Enterprise Edition.
https://influxdata.com/blog/update-on-influxdb-clustering-high-availability-and-monetization/
Within the next few years, we will hopefully have a well-engineered time-series database that is handling all the hard problems relating to databases: replication, failover, data safety, scalability, backup...
At the moment, there is no silver bullet.
What to do
Evaluate the volume of data to be expected.
100 metrics * 100 sources * 1 second => 10000 datapoints per second => 864 Mega-datapoints per day.
The nice thing about times series databases is that they use a compact format, they compress well, they aggregate datapoints, and they clean old data. (Plus they come with features relevant to time data series.)
Supposing that a datapoint is treated as 4 bytes, that's only a few Gigabytes per day. Lucky for us, there are systems with 10 cores and 10 TB drives readily available. That could probably run on a single node.
The alternative is to use a classic NoSQL database (Cassandra, ElasticSearch or Riak) then engineer the missing bits in the application. These databases may not be optimized for that kind of storage (or are they? modern databases are so complex and optimized, can't know for sure unless benchmarked).
You should evaluate the capacity required by your application. Write a proof of concept with these various databases and measures things.
See if it falls within the limitations of InfluxDB. If so, it's probably the best bet. If not, you'll have to make your own solution on top of something else.
I’ve recently come across InfluxDB in this sub and it looks really interesting. What do you use it for in terms of APIs? Is logging/monitoring a popular use case?
Hi,
maybe this has been discussed, but I am new to both systems and quite frankly I am overwhelmed by the different options.
So here is the situation:
We have an influxdb v2 where data about the internet usage is stored for example. Now we want to store the data in Prometheus too.
I have seen the influxdb exporter and a native api option. But it's really confusing. Please help me find the best way to do this.