Troubleshooting Elasticsearch 6.8 Metadata Generation Issue in DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> For getting elasticsearch 6.8 to work where it will pull from it but generate no metadata? How do I fix this?

2

To address the issue of Elasticsearch 6.8 not generating metadata in DataHub, you can follow these steps:

  1. Check Elasticsearch Indices: Ensure that the necessary indices are present in your Elasticsearch instance. You can do this by running the following command:

    curl http://<your-elasticsearch-endpoint>/_cat/indices?v

    Look for indices like datasetindex_v2, corpuserindex_v2, etc. If these indices are missing, it could be a reason for the lack of metadata generation.

  2. Elasticsearch Compatibility: According to <https://github.com/datahub-project/datahub/issues/1722|this GitHub issue>, Elasticsearch 6.x should be compatible with DataHub, but there might be issues with specific configurations. Ensure that your Elasticsearch setup is correctly configured to work with DataHub.

  3. DataHub Upgrade: If the indices are missing or not functioning correctly, you might need to run the datahub-upgrade component. This component is responsible for generating all the necessary indices in Elasticsearch. You can refer to the <https://datahubproject.io/docs/how/restore-indices/|DataHub documentation> for more details on restoring indices.

  4. Check Network Configuration: Ensure that your DataHub instance can communicate with the Elasticsearch service without any network issues. Network routing issues can cause failures in index creation.

  5. Verify Data in Database: Confirm that the metadata is still present in your database (e.g., MySQL, PostgreSQL). You can do this by querying the relevant tables directly.

Would you like more detailed steps on any of these processes?

Sources:

3

Hi <@U08JWQ3T0M6>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

4

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><