Incorrect understanding
I think there was a mistake in my understanding of how labeling in prometheus works. My incorrect understanding was:
- before applying
regex, string would be first split onseparator(otherwise what is its purpose?), - each substring has
regexevaluated against it, - if match groups are declared and found, they will be available as indexed values available to use in
target_labelandreplacementfields. - if
regexdoes not match, then that substring will be ignored. - because
regexis expected to be applied to each substring after the split, it will lead to multiple labels from multiple substrings.
Correct understanding
However, from brian-brazil's post linked in his answer and Prometheus's documentation, it seems the following happening:
- All
__metatags are combined into one longseparatorseparated line. regexis applied on that line only once.- If
regexmatches and includes groups, they are indexed beginning from 1 and available for use intarget_labelandreplacement. separatorseems to be getting ignored in this section even if you mention it.
Config from corrected understanding
From this idea and following from example in the question, I was able to make the following config that works
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,a=([a-z0-9_]+),.+'
target_label: 'a'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,b=([a-z0-9_]+),.+'
target_label: 'b'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,c=([a-z0-9_]+),.+'
target_label: 'c'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,d=([a-z0-9_]+),.+'
target_label: 'd'
replacement: ${1}
Caveats
I believe both approaches (the approach brian-brazil wrote in his blogpost, and what I am using above) have caveats - we either need to know all the labels we want beforehand, or have a set number of them. This means if a developer wants to associate different, or more labels with his/her service, s/he would need to work with ops as general flow will not be able to handle it. I think it is a minor caveat that should be addressed.
https://www.robustperception.io/extracting-full-labels-from-consul-tags/ shows how to do this, in particular the last example.