You have to use a Gauge and not a Counter.
Example from the README:
from prometheus_client import Gauge
g = Gauge('my_inprogress_requests', 'Description of gauge')
g.inc() # Increment by 1
g.dec(10) # Decrement by given value
g.set(4.2) # Set to a given value
See also Prometheus metric types: https://prometheus.io/docs/concepts/metric_types/
Answer from svenwltr on Stack OverflowCounter
A counter is a cumulative metric that represents a single numerical value that only ever goes up. A counter is typically used to count requests served, tasks completed, errors occurred, etc. Counters should not be used to expose current counts of items whose number can also go down, e.g. the number of currently running goroutines. Use gauges for this use case.
Gauge
A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
Gauges are typically used for measured values like temperatures or current memory usage, but also "counts" that can go up and down, like the number of running goroutines.
Set Prometheus Counter value to 0 - Stack Overflow
Allow setting counter value when exposing existing counts
go - Prometheus counters: How to get current value with golang client? - Stack Overflow
How to take the previous counter value and add it to the new one in case it resets to zero on Grafana?
You have to use a Gauge and not a Counter.
Example from the README:
from prometheus_client import Gauge
g = Gauge('my_inprogress_requests', 'Description of gauge')
g.inc() # Increment by 1
g.dec(10) # Decrement by given value
g.set(4.2) # Set to a given value
See also Prometheus metric types: https://prometheus.io/docs/concepts/metric_types/
Counter
A counter is a cumulative metric that represents a single numerical value that only ever goes up. A counter is typically used to count requests served, tasks completed, errors occurred, etc. Counters should not be used to expose current counts of items whose number can also go down, e.g. the number of currently running goroutines. Use gauges for this use case.
Gauge
A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
Gauges are typically used for measured values like temperatures or current memory usage, but also "counts" that can go up and down, like the number of running goroutines.
Counter value can be set by accessing private _value property. I strongly recommend avoiding this solution, unless you know for sure that the values you are going to set will only increase and not decrease. Otherwise use Gauge instead.
from prometheus_client import Counter
c = Counter("foo", "bar")
c._value.set(50)
c._value.get()
50
c._value.set(101)
c._value.get()
101
# the same with labels
cd = Counter("bar", "foo", labelnames=['name'])
cd.labels(name="spam")._value.set(20)
What can go wrong with this approach
The main problem is the counter reset. Whenever PromQL functions (such as rate(), increase()) detect a counter decrease, it is considered as a reset. Consider this simple example:
my_metric 1 2 3 4 3
increase(my_metric) 0 1 1 1 3
So if you decrease a counter value, some functions will treat it as if counter became 0 and then instantly went up to whichever value it is now. This will ruin all calculations on that counter, especially if it's a big one (like sent bytes, requests handled, etc).
It's easy, have a function to fetch Prometheus counter value
import (
"github.com/prometheus/client_golang/prometheus"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/log"
)
func GetCounterValue(metric *prometheus.CounterVec) float64 {
var m = &dto.Metric{}
if err := metric.WithLabelValues("label1", "label2").Write(m); err != nil {
log.Error(err)
return 0
}
return m.Counter.GetValue()
}
Currently there is no way to get the value of a counter in the official Golang implementation.
You can also avoid double counting by incrementing your own counter and use an CounterFunc to collect it.
Note: use integral type and atomic to avoid concurrent access issues
// declare the counter as unsigned int
var requestsCounter uint64 = 0
// register counter in Prometheus collector
prometheus.MustRegister(prometheus.NewCounterFunc(
prometheus.CounterOpts{
Name: "requests_total",
Help: "Counts number of requests",
},
func() float64 {
return float64(atomic.LoadUint64(&requestsCounter))
}))
// somewhere in your code
atomic.AddUint64(&requestsCounter, 1)
So I have a counter metric, I aggregate it by sum based on two label values. My question is, after the application restarts the counter is going to reset to zero, but on grafana I want to keep the counter persistent, meaning that when the counter becomes zero, I want to take the previous value and add it to the new counter value.
So if counter metric is 5.0, application restarts and now the counter metric is 0, I basically want to take previous value 5 and add it to the current value 0.
Does this make sense? I don't know how to do it.