Troubleshooting DataHub System Update Job failures related to Elasticsearch reindexing

troubleshoot
1

Original Slack Thread

Hello. After some upgrades of DataHub installed through Helm Chart, DataHub System Update Job fails with this error in a loop:

2023-10-13 12:51:22,969 [main] WARN  c.l.m.s.e.i.ESIndexBuilder:296 - Task: s_IASc-mSxC5xuRa-PFEkw:2945941 - Document counts do not match 58 != 0. Complete: 0.0%```
A reset of ES solves the issue but then issue is back after few upgrades. Issue happens without using DataHub.
Are you aware of a know bug ? Can it be related to DataHub System Update Job running in parallel of DataHub GMS after initial install ?
Thanks :slightly_smiling_face:
2

This can occur when ES cannot reindex documents from one index to another. Sometimes it is one or two documents off which could mean some bad data or something added a new document during index. In this case it is all of the documents so the error is on the ES side. See if you can pull any additional information on the task from ES’s task API documented here: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/tasks.html