Troubleshooting metadata deletion and cache clearing in DataHub v0.12.0

troubleshoot
1

Original Slack Thread

Hi team!

I have a problem with metadata deletion. I have deleted some datasets in schema recursively and then schema with database itself using commands:

datahub delete --urn "urn:li:container:110abe4687ad3218923879f98b8dfd0a" --hard
datahub delete --urn "urn:li:container:dc49a9c896de1b2e731d8dbea7dca543" --hard```
But some data related to them still remain in the Navigate UI.

Also I deleted some datasets using match query:
```datahub delete --entity-type dataset --query "target" --platform elasticsearch --hard```
Datasets were deleted but datasets count in Navigate UI remains the same, and when I try to click on data source empty pages without any data appear.

Could you tell me please if I did something wrong and how can I fix this issue. Maybe I should take any additional steps?![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06CM4APRK9/image.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06CM4SQWKZ/image.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

I’m using DataHub v0.12.0

4

<@U03CS6YECF3> I think it has some latency to get updated but <@U03BEML16LB> might know more about how this works

5

might be counts are cached. I think clearing the caches of ES indices and a restart of GMS would fix the issue.

6

<@U0445MUD81W> thank you. I’ve cleared ES indices cache and restarted GMS, but the problem still persists

7

Also after restart it seems that I can’t add new users and no metadata appears after injestion
<@UV14447EU> <@U03BEML16LB>
Do you know the way I can fix it?

8

Also I noticed that injestion pipelines don’t run on schedule anymore

9

Do you see anything in the logs?

10

This is gms logsattachment

11

And this is frontend logsattachment

12

I am having same problem of deleted metadata appearing. <@U0445MUD81W> How do you clear the cache of ES indices?

13

it looks like it is not solving the problem. ES cache invalidation after a certain interval, you may have to clear the cache manually only when we want to see the changes immediately. Any here the curl for clear ES indices cache might be useful for some other use cases.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clearcache.html

14

<@UV14447EU> Do you have any suggestions according to my logs?

15

Did you stopped DataHub or why I can see a termination in the logs?

,nbxioftb5qma7ru0,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,"2024-01-10 13:33:34.117:INFO:oejsh.ContextHandler:JettyShutdownThread: Stopped o.e.j.w.WebAppContext@6574a52c{Open source GMS,/,null,STOPPED}{file:///datahub/datahub-gms/bin/war.war}",2024-01-10T13:33:34.117408532Z
,2bmaum9nlygq8uyn,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,2024-01-10 13:33:33.748:INFO:oejshC.ROOT:JettyShutdownThread: Closing Spring root WebApplicationContext,2024-01-10T13:33:33.748359310Z
,5ub39i03zd9ao5oq,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,2024-01-10 13:33:33.742:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet 'apiServlet',2024-01-10T13:33:33.742310698Z
,gqeoa25cvftuo49t,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,2024-01-10 13:33:33.740:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet 'authServlet',2024-01-10T13:33:33.740559741Z
,klacninjsxbtoo7y,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,2024-01-10 13:33:33.736:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet 'openapiServlet',2024-01-10T13:33:33.736596073Z
,wrody45tt0dzxf5o,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,2024-01-10 13:33:33.729:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet 'schemaRegistryServlet',2024-01-10T13:33:33.729937819Z
,1zlgfn8azsk5fze0,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,2024-01-10 13:33:33.728:INFO:oejs.session:JettyShutdownThread: node0 Stopped scavenging,2024-01-10T13:33:33.728203591Z
,c24i38wwo0utbbj9,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,"2024-01-10 13:33:33.727:INFO:oejs.AbstractConnector:JettyShutdownThread: Stopped ServerConnector@44821a96{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}",2024-01-10T13:33:33.728013231Z
,p59lctuz7y36zp9v,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,754bd8886f,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:33:38.532283203Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-754bd8886f-4btpg,phonic-heaven-126923,k8s_container,ERROR,2024/01/10 13:33:33 Received signal: terminated,2024-01-10T13:33:33.715273900Z
,wlee4ptbsncyu69e,gke-infra-n2-16-8deb7373-8pcu,datahub,datahub-gms,7b8cb694d7,projects/phonic-heaven-126923/logs/stderr,,2024-01-10T13:32:13.638086178Z,infra,datahub-gms,us-central1-c,datahub-blue,datahub-datahub-gms-7b8cb694d7-grthd,phonic-heaven-126923,k8s_container,ERROR,2024-01-10 13:32:08.996:INFO:oejs.Server:main: Started @35095ms,2024-01-10T13:32:08.997153306Z```
16

<@UV14447EU> Maybe it appeared because I had restarted datahub that time.
Sending you the most actual logs for now

https://files.slack.com/files-pri/TUMKD5EGJ-F06DMS7AP27/datahub-gms-downloaded-logs-20240112-125429.csv?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c(image larger than 4 MB)
attachment

17

Hello <@UV14447EU> Has anything become clearer with the new logs?

18

hey gang, we just talked on the marathon IRL but this situation would definitely arise if the deleted containers and datasets were removed from the database but the elastic indexes were not updated

19

The latest logs indicates a different issue with GMS not being able to connect to either sql or elastisearch

20

<@U03MF8MU5P0> <@U03BEML16LB> Thanks for your assistance yesterday. I had contacted our DevOps team, and they said that checked the logs and did not find any that indicate such problems. They mentioned that these logs were generated when the entire system was restarted and that it was normal that there was no access there temporarily. I’m attaching the latest current logs, maybe they can tell more about the issue.

https://files.slack.com/files-pri/TUMKD5EGJ-F06EG6FEZBQ/datahub-gms-downloaded-logs-20240119-112740.csv?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c(image larger than 4 MB)
attachmentattachmentattachment