Is there a trick to the ES helm chart on increasing shards?
Hey there! 
Make sure your message includes the following information if relevant, so we can help more effectively!
-
Which DataHub version are you using? (e.g. 0.12.0)
-
How are you deploying DataHub? (e.g. Helm, Quickstart, etc)
v 11
Under elasticsearch.index.settingsOverrides you can set the number of shards: https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L357
the issue is i’m getting errors that i’m at 3000/3000 shards, I can just re-allocate shards to the indexes that are needed?
Oh, understood, you’re deploying with the prerequisites ES community helm chart? You can change your cluster soft shard limit with the cluster settings described here: https://www.elastic.co/guide/en/elasticsearch/reference/7.1/misc-cluster.html#cluster-shard-limit but it seems extremely odd that you would have 3k shards unless you’re utilizing the cluster for more than just DataHub. It should not be getting anywhere near that limit. I would definitely look into what is causing you to have that many shards utilized before just increasing the limit.
its… just being used for data hub, most likely the timeseries data/stateful?
though we only do table level profiling
we currently run about 5.5k datasets
Document count wouldn’t affect number of shards, just what indices are there would. You likely need to clean up some indices that shouldn’t be there, not sure why you would have enough to hit the shard limit.