Addressing Elasticsearch Warnings in Datahub Helm Charts

troubleshoot
1

Original Slack Thread

Hi all,
Our GMS pod is continuously logging following warnings from Elasticsearch:
(We are running Datahub version v0.10.5 in EKS)

[299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. 
See <https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html> to enable security."],

[299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. 
Consider cold or frozen tiers in place of frozen indices."]```
Anyone had the same? Any ideas how to:
• Enable the "_built-in security features_" in ES? Or what could be the way in Datahub Helm charts to fix this?
• Fix deprecated "_frozen indices"_ warning_,_ and use "_cold or frozen tiers_" instead?

We have 3 replicas of ES running.
Our configs in values.yaml are following for global.elasticsearch:
```....
global:
  elasticsearch:
    host: "elasticsearch-master"
    port: "9200"
    skipcheck: "false"
    insecure: "false"
....```
2

• This warning indicates you have the security plugin enabled but it is not configured, if you don’t want security you can disable xpack xpack.security.enabled: false to elasticsearch.yml
• For frozen indices warning due to, the ignore_throttled parameter got <https://github.com/elastic/elasticsearch/pull/77479|deprecated in ES 7.16>
on solution is to upgrade the elatsicseach client in Datahub, other solution is to use an older version of ES or neglect this warning for now

3

Thank you <@U0445MUD81W>
Regarding to the security plugin, still not quite sure how to configure it with Helm Charts for kubernetes deployment.
Should it be provided as env variable, like:

  elasticsearch:
    ...
    xpack.security.enabled: "true"```
Or?

Where it gets assigned in <https://github.com/acryldata/datahub-helm/blob/datahub-0.2.181/charts/datahub/templates/elasticsearch-setup-job.yml|here>?
4

this configuration is set to the elasticsearch service, so you have to set these values in elasticsearch.yml before you provision or edit after you provision ES.
I’m not that good at Helm charts, this value should when setting prerequisites for datahub like Mysql, Kafka, and ES
I think this might the file
https://github.com/acryldata/datahub-helm/blob/datahub-0.2.181/charts/prerequisites/values.yaml

5

please take look the blog it may helps you in figuring out

https://copyprogramming.com/howto/how-to-disable-xpack-security-enabled-from-helm-char-in-elasticsearch-deployment-on-k8s

6

An easy solution, I can suggest if you have access to the elasticsearch service, just edit elasticsearch.yml and restart the service

7

Thanks for the help <@U0445MUD81W>! I’ll take a look on that :+1: If not able to find a way to do it from Helm Chart, I guess accessing the ES service after the deployment and configuring it there would be the way to go then

8

I was just searching for how to do the same, adding this to the elasticsearch configuration in the prereqs chart values.yaml looks to resolve the warning spam:

  enabled: true  

  extraEnvs:
    - name: xpack.security.enabled
      value: "false"```
Hopefully useful for anyone else who comes across this.