The way I sorted this problem was doing the following prometheus query for Grafana. This query allows me to identify the total_processing_time consumed by an individual customer across all PODs in a K8s cluster within a given time range. The catch is that this query handles the following corner cases:

• If the querying time_range encompasses only the last 1h but there are older timeseries that were started before that, the query below "offsets to zero" these timeseries by removing the value of the first measurement point within the time_range. This way these series are considered as if they have started within the time range.

• If a new timeseries (due to a new POD) show up within the querying time_range, and the first measurement value is greater than 0, then the first measurement value of the new timeseries is added to the calculation to fulfill the gap from the increment function.

sum(
    # Calculates the difference between the last measurement 
    # and the first measurement in the given time_range for each 
    # timeseries, this helps identify how much additional resources
    # were consumed in the given time_range
    increase(
        total_processing_time{customer_id="$customer_id"}[$__range]
    )
    # Adds to the difference the minimum value of each timeseries, 
    # this helps to account for the timeseries that started within 
    # the given time_range with the first measurement greater than 0
    # (not considered by the previous increase function)
    + min_over_time(
        total_processing_time{customer_id="$customer_id"}[$__range]
    )
)
# Removes the timeseries measurements being carried over from before 
# the start of the time_range, this offsets the running timeseries
# as if they had started from zero within the time_range
- sum(
    total_processing_time{customer_id="$customer_id"} @ start() or vector(0)
)

There's no increases there, the time series appear with the value 1. If they had increased from 0 to 1 rather than nothing to 1 then increase would show a non-zero value. I'd recommend initialising your metrics with the label values you know about.

When prometheus is evaluating an expression, the operation implicitly apply to metric that share identical set of labels.

Despite the fact I specified the metric name and most labels, Prometheus was looking for metrics that have the same set of labels.

However, in this case, two metrics have different label values, so they can't match ! (one metric has metric="Used"the other has metric="Total". It could be that one of the metrics has some extra labels).

The solution is to use ignore (or on) to reduce the set of considered labels:

fs_bytes{instance="localhost:9108",metric="Used"} / ignoring(metric) fs_bytes{instance="localhost:9108",metric="Total"}

Read the fine manual ! (here)

I think your can do some kind of alerting on a metric rate with something like this:

ALERT DropInMetricsFromExporter
  IF rate(<metric_name>[1m]) == 0
  FOR 3m
  ANNOTATIONS {
    summary = "Rate of metrics is 0 {{ $labels.<your_label> }}",
    description = "Rate of metric dropped, actually: {{ $value }}%",
}

The main idea is to alert whenever the metric rate is at 0 for 3 minutes, with the proper metric name and a label somewhere telling from which exporter it comes it should give you the correct information.

Choosing the right metric to monitor by exporter could be complex, without more insight is hard to give a better advice out of vacuum.
This blog post could be an inspiration also for a more generic detection.