Prometheus, like Graphite, is a time-series storage engine.
Grafana can then query Prometheus to generate graphics and alerts.
https://prometheus.io/docs/introduction/faq/
As the documentation cites, Prometheus, unlike other metrics storage systems, uses a (debatable) "pull" model.
This means that there is a (stand-alone) Prometheus server that must be downloaded/installed. This server then periodically makes HTTP GET requests (pull) to a list of application servers - such as a Java SpringBoot server to fetch (in-memory) stored metrics.
Ref: https://prometheus.io/docs/introduction/faq/#why-do-you-pull-rather-than-push?
Thus the (Spring Boot) application must expose a metrics end-point that the Prometheus server can pull from (default is /metrics).
Ref: https://github.com/prometheus/client_java
Thus there is much documentation available on Google but that is the (arguably convoluted) topology - along with arguments from the SoundCloud and Prometheus folks as to why a "pull" model is preferred over "push" as every other metrics framework employs.
Answer from Darrell Teague on Stack Overflowmaven - How to expose metrics to Prometheus from a Java (Spring boot) application - Stack Overflow
python - Expose package metrics to Prometheus with prometheus_client - Stack Overflow
Best way to expose custom metrics to Prometheus for a kubernetes cron job
How to make docker container be able to communicate with my host's localhost
Prometheus, like Graphite, is a time-series storage engine.
Grafana can then query Prometheus to generate graphics and alerts.
https://prometheus.io/docs/introduction/faq/
As the documentation cites, Prometheus, unlike other metrics storage systems, uses a (debatable) "pull" model.
This means that there is a (stand-alone) Prometheus server that must be downloaded/installed. This server then periodically makes HTTP GET requests (pull) to a list of application servers - such as a Java SpringBoot server to fetch (in-memory) stored metrics.
Ref: https://prometheus.io/docs/introduction/faq/#why-do-you-pull-rather-than-push?
Thus the (Spring Boot) application must expose a metrics end-point that the Prometheus server can pull from (default is /metrics).
Ref: https://github.com/prometheus/client_java
Thus there is much documentation available on Google but that is the (arguably convoluted) topology - along with arguments from the SoundCloud and Prometheus folks as to why a "pull" model is preferred over "push" as every other metrics framework employs.
For Intergrating Prometheus, add the following dependencies in your POM.XML
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_spring_boot</artifactId>
<version>0.1.0</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>0.1.0</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.1.0</version>
</dependency>
In your SpringBoot Application Class, add the Annonation @EnablePrometheusEndpoint
In your Controller, you can define a Custom Counter
private static final Counter myCounter = Counter.build()
.name("CounterName")
.labelNames("status")
.help("Counter desc").register();
In your service, you can have appropriate logic for your Counter which would be automatically pulled by Prometheus
@RequestMapping("/myService")
public void endpoint() {
String processingResult = yourLogic(myCounter);
myCounter.labels("label1",processingResult).inc();
}
I have a kubernetes cron job that is relatively short lived (a few minutes). Through this cron job I expose to the prometheus scrapper a couple of custom metrics that encode the timestamp of the most recent edit of a file.
I then use these metrics to create alerts (alert triggers if time() - timestamp > 86400).
I realized that after the cronjob ends the metrics disappear which may affect alerting. So I researched the potential solutions. One seems to be to push the metrics to PushGateway and the other to have a sidecar-type of permanent kubernetes service that would just keep the prometheus HTTP server running to expose and update the metrics continually.
Is there a solution more preferable than the other? What is considered better practice?

