Troubleshooting Datahub Ingestion Discrepancy in Postgres Integration

ingestion
1

Original Slack Thread

Hi, My Datahub Ingestion recipe for Postgres ingestion is successful and shows 294 assets but the assets that is shown in the ingestion I am not able to find it in Datahub. Where wopuld the problem be?

This is in Datahub Version 0.11.0

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

I am using UI for the ingestion.
Version - Datahub version - 0.11.0
Data Source - Postgres

4

Many of these events are available but not discoverable

5

these MCPs are not indexing in Elasticseach due to some reason, check your elastic search health and mce & mae consumers are running fine.

6

Hi Arun, yes please post any relevant logs here for us to assess more effectively :slightly_smiling_face:

7

Thanks guys…

I see in the ingest log that the ingest is succeding…

2024-03-27 21:08:19,037 [qtp2141817446-10493] INFO  c.l.m.entity.EntityServiceImpl:1835 - Ingesting aspect with name browsePathsV2, urn urn:li:dataset:(urn:li:dataPlatform:postgres,magichat.peloton.public.pg_stat_statements,PROD)```
I see that the elastic search cluster is also Green and stable.

Do i have to set `SHOW_BROWSE_V2` to true - <https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml#L112C21-L112C35>

As of now that is not set as part of the GMS, will that help?
8

Thanks Arun! I believe <@U03MF8MU5P0> should be able to assist with this :slightly_smiling_face:

9

The log line shown only indicates that GMS has received the request. If the documents are not present in Elasticsearch, the next step is to try to restore ES index documents from SQL. Please see restoreIndices documentation here: https://datahubproject.io/docs/api/restli/restore-indices/ and here https://datahubproject.io/docs/how/restore-indices/. The First one is a rest endpoint which you can run a single urn to see if it works. Then the second one includes directions on running a job to restore the rest.