Discussion on the Relationship Between `ELASTICSEARCH_QUERY_MAX_TERM_BUCKET_SIZE` and `searchFlags.maxAggValues` Parameters

troubleshoot
1

Original Slack Thread

Hello, I have a question related to some things introduced by 0.10.x version of Datahub. I recently got hurt by searchAcrossEntities facets (buckets) count limited to 20. I found 2 related settings to this:

  1. Environment variable ELASTICSEARCH_QUERY_MAX_TERM_BUCKET_SIZE
  2. searchFlags.maxAggValues input parameter for the query
    What I have noticed is that changing value of ELASTICSEARCH_QUERY_MAX_TERM_BUCKET_SIZE does not change actual limit of returned bucket count, while changing searchFlags.maxAggValues (at least in version 0.10.5 ) actually changes the bucket count limit. I am a bit confused what is intended relation between the env variable and the query input parameter?
    This puzzles me especially, considering the query builder:
    https://github.com/datahub-project/datahub/blob/master/metadata-io/src/main/java/com/linkedin/metadata/search/elasticsearch/query/request/SearchRequestHandler.java#L200|https://github.com/datahub-project/datahub/blob/master/metadata-io/src/main/java/c[…]ta/search/elasticsearch/query/request/SearchRequestHandler.java
    Does not use finalSearchFlags when building aggregations as well as does not seem to use parameters coming from ELASTICSEARCH_QUERY_MAX_TERM_BUCKET_SIZE . What am I missing here?
2

<@U03MF8MU5P0> might be able to speak to this!

3

There is possibly one or more bugs here, this is actually the first time I’ve seen where more aggregates are needed. I think the intent here is to set an upper limit with the env variable and not set a default value. I believe that somewhere there is a check to validate the searchFlags in the request against this upper limit. I do believe the finalSearchFlags should be respected here

4

The GQL variable ‘searchFlags.maxAggValues’ [https://datahubproject.io/docs/0.10.5/graphql/inputObjects#searchflags] should change the bucket count limit. However, it’s not working as expected.
Any solution here <@U03MF8MU5P0> ?