Handling NPE Errors in DBT Ingestions and Snowflake Jobs in DataHub

ingestion
1

Original Slack Thread

Hi team
Hello, this is the error message I am getting on dbt Ingestions (DH version 11.0).
<@U04N9PYJBEW> Thanks for helping me out on this. This is the error from the Office Hours call just now.

              {'error': 'Unable to emit metadata to DataHub GMS: java.lang.NullPointerException',
               'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
                        'message': 'java.lang.NullPointerException',
                        'status': 500,
                        'id': 'urn:li:dataset:```
2

<@U01GZEETMEZ> have you seen a NPE from the rest sink before? Could we be producing invalid aspects as part of dbt ingestion?

3

Checking… What would I look for regarding an invalid aspect on a dbt ingestion?

4

Ah I’m mainly asking Harshal

5

Sure, I am also digging in some more on my side. Does this help?

attachment
attachment1064×226 17.5 KB
attachment
attachment1441×203 31.2 KB

6

I haven’t seen that before. Do you have the corresponding GMS logs handy?

7

Yes, I can get those.

8

https://files.slack.com/files-pri/TUMKD5EGJ-F05UKKA2L3H/gms-logs-20231003?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c(image larger than 4 MB)

9

Thanks <@U01GZEETMEZ>

10

This line looks suspicious

java.io.IOException: Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=<https://vpc-datahub-2frsptdmsahrwqd5lhurzzx4d4.us-west-2.es.amazonaws.com:443>, response=HTTP/1.1 200 OK}```
Is it possible that ES is unhealthy?

I suspect that this is a more general problem in the backend, and not something specific to dbt because it looks like it's failing on both the subtypes aspect and the main MCEs

In any case, the NPE definitely should be handled better, although I'm not exactly sure where it's originating from, since it seems unlikely that there's a bug in `RestliUtil.toTask` cc <@UV5UEC3LN>
11

Hmm, we did just add a second node to our OpenSearch cluster in dev. I don’t see any alarm’s either. I will double check.

12

This is not only dbt, we are receiving similar error for the few of Snowflake jobs for certain schema, while all other schema runs fine.

13 
java.net.URISyntaxException: mismatched paren nesting: urn:li:dataset:(urn:li:dataPlatform:snowflake,dev_mart_db.ea_ops_trans.sharp_ai_conversations_dr_predict```
This also seems problematic
14

There are tons of these ^

15

Yes, do you think this is an issue with DH or dbt?

16

Not sure, we haven’t seen this on other instances that are using DBT ingestion to my knowledge though

17

It’s also happening in snowflake I believe. I’ll confirm

18

The one above in the error message is a snowflake dataset so that checks out

19

The above one is snowflake dataset but through dbt source
We do have similar issue, when directly ingesting snowflake. Thats the weird part, it gives us error on two of the schemas and everything else runs smooth, thats where i thought its source side issue over DH issue. But i am not so sure.

20

Here are some more clues. We are seeing a NPE when ingesting DBT, however when searching DataHub for the Snowflake object I also get a 500 Error on the UI.

Getting the following error on a dbt ingestion:

And when I pull up that same table in DH, I also get an error:

attachment
attachment712×338 11 KB
attachmentattachment