Setting Up OpenSearch Nodes with DataHub Version 2.11.0

ingestion
1

Original Slack Thread

This message was deleted.

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

To connect multiple nodes using OpenSearch with DataHub version 2.11.0, you need to ensure that your OpenSearch setup is compatible and properly configured with DataHub. Here are some steps and considerations based on the available data sources:

  1. Compatibility: Ensure that the version of OpenSearch you are using is compatible with DataHub. According to <Slack Slack Discussion>, DataHub 0.13.0 has been tested with OpenSearch 2.9 and 2.11. While there is no explicit mention of DataHub 2.11.0, it is likely that similar configurations apply.

  2. Configuration: You need to configure your values.yaml file to point to your OpenSearch cluster. This involves setting the host, port, and any necessary authentication details. For example, as per the <Deploying to AWS | DataHub Deployment Guide>, you should configure the OpenSearch settings under the elasticsearch section in your values.yaml file.

  3. Security and Authentication: If you have fine-grained access control enabled, you may need to create Kubernetes secrets for authentication. This is detailed in the <Deploying to AWS | DataHub Deployment Guide>.

  4. Environment Variables: Ensure that the USE_AWS_ELASTICSEARCH environment variable is set to true if you are using AWS OpenSearch, as mentioned in the <Deploying to AWS | DataHub Deployment Guide>.

  5. Cluster Setup: Make sure your OpenSearch cluster is properly set up to handle multiple nodes. This includes configuring the cluster settings to allow for node discovery and communication.

Would you like more detailed steps on any specific part of this process, such as configuring the values.yaml file or setting up Kubernetes secrets?

Sources: