Query too long

[abursavich]

What happened?:

The query that matches all ready pods in a large deployment is being rejected by VictoriaMetrics.

E0703 20:49:59.724673 1 provider.go:150] unable to fetch metrics from prometheus: 422: too long query; got 25852 bytes; mustn't exceed -search.maxQueryLen=16384 bytes

We can (and will for now) increase the maximum size of a query that VictoriaMetrics allows, but we could also reduce the size of the query that prometheus-adapter constructs.

Given these pods:

1. podfromareplicasetwithalongname-24vm7
2. podfromareplicasetwithalongname-4h7zn
3. podfromareplicasetwithalongname-gpmg2
4. podfromareplicasetwithalongname-scgch
5. podfromareplicasetwithalongname-x7ks6

The constructed query is:
some_metric_name{namespace="foobar",pod=~"podfromareplicasetwithalongname-24vm7|podfromareplicasetwithalongname-4h7zn|podfromareplicasetwithalongname-gpmg2|podfromareplicasetwithalongname-scgch|podfromareplicasetwithalongname-x7ks6"}

But it could be:
some_metric_name{namespace="foobar",pod=~""podfromareplicasetwithalongname-(24vm7|4h7zn|gpmg2|scgch|x7ks6)"}

Now imagine that with 1500 pods instead of 5.

What did you expect to happen?:

Prometheus-adapter should make a reasonable effort to create small queries.

[dgrisonnet]

/triage accepted
/assign @abursavich

[YZ775]

Hello.
We encountered the same problem with a namespace which has a lot of pods.

When we execute the command kubectl top pod -n <namespace> in such a namespace, it returns error error: Metrics not available for pod foo/bar, age: 4h48m35.769652691s
and prometheus-adapter outputs too long query error.

If there are thousands of nodes in the cluster, DaemonSet will be created for each node, so it will make a lot of pods in the same namespace.

The PR to fix this problem is created but closed automatically.
Are there any plans to reopen the PR?

• pkg/naming: shorten regexp when matching many similar names #670

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted