For now the only solution I have is with query duplication:

(
  (sum(kafka_consumergroup_lag[1d]) by (consumergroup) >= 100)
  * on(consumergroup) group_left(team)
  catalog_entities_info
)
or ignoring(team) (sum(kafka_consumergroup_lag[1d]) by (consumergroup) >= 100)

First half will add team label where possible, and second or half will add missed data (important to merge vectors while ignoring added team label, otherwise we will have duplicates).

You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

The important part here is the operation + on(service_name,node) group_left(env):

• the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
• group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)

I think the keyword in that comment is shareable or in other words reusable rules. Meaning you (often) preserve more labels while using ignoring compared to on and the results will be (usually) a rule with more of it's original labels left intact, so it can be reused for more scenarios.

Imagine these time series:

instance_cpu_time_ns{app="lion", proc="web", rev="34d0f99", env="prod", job="cluster-manager"}
instance_cpu_time_ns{app="elephant", proc="worker", rev="34d0f99", env="prod", job="cluster-manager"}
instance_cpu_time_ns{app="turtle", proc="api", rev="4d3a513", env="prod", job="cluster-manager"}
instance_cpu_time_ns{app="fox", proc="widget", rev="4d3a513", env="prod", job="cluster-manager"}
...

A query with ignoring(rev) leaves out all the other labels in the result, compared with the same query with on(app).

But the result of on and ignoring would be identical if you use them with mutually exclusive set of labels, like the example you are mentioning.

The RHS has no instance label, so it's trying to match all those series to one on the LHS. Try max by (node, instance) (kube_node_labels{label_grid="true"})

It is OK to use group_left() for 1:1 query if you need retaining all the labels from time series on the left side (or from the right side if group_right() is used). This doesn't introduce performance penalty.

If you put a list of labels inside group_left(), then these labels will be copied from time series on the right side additionally to all the labels from time series on the left side of the specified binary operator.

See more details in the official docs.

If you need retaining only some labels from the left side, then just drop unneeded labels after the calculations of q1 * on (...) group_left() q2. Prometheus doesn't provide functions, which can drop unneeded labels from time series (or leave only the needed labels), but this functionality can be emulated with sum(...) without (...) or sum(...) by (...) in most cases. For example, the following query leaves only (namespace, instance_id, label2) labels from time series matching metric1{namespace="ns1"} in the result:

sum(
  (metric1{namespace="ns1"} == 1)
    * on (namespace, instance_id) group_left()
  (metric2{namespace="ns1", label3="something"} == 1)
) by (namespace, instance_id, label2)

Note that the sum() may return unexpected results if multiple time series have the same set of output labels specified inside by (...).

P.S. Take a look also at label_del and label_keep functions provided by VictoriaMetrics - the Prometheus-like monitoring solution I work on. These functions are easier to use instead of sum(...) by (...) or sum(...) without (...) trick when some labels need to be dropped / left in the result.