Troubleshooting datahub errors and schema registry configuration in Kubernetes deployment using Helm charts

all-things-deployment
1

Original Slack Thread

Hello,

I have been tasked to debug errors messages appearing on our datahub deployment on kubernetes using the helm chart provided by acryldata bumped chart version from 0.2.181 to 0.2.182
We are ingesting metadata from Databricks, Sagemaker, dbt. When displaying the logs of these 3 jobs, I get recurring similar errors being :
• datahub-gms return status 500 (see picture)
• Error registering Avro Schema
Even after trying the solution propsed in <Issues · acryldata/datahub-helm · GitHub issue> the problem keeps recurring.

Thank you for the help!attachment

2

<@U04UKA5L5LK> might be able to speak to this!

3

Hi, what schema registry are you using?

4

I think it is the provided by the helm charts version 0.2.182

5

There is a known issue with internal and glue schema registries in the last two versions of datahub, which could cause this issue, can you make sure you are using the confluent schema registry?

6

so I need to change the schema registry image in the values.yaml file? https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml

7

lines 487 to 498

8

Yes! That’s right. Sorry, I think we had a different chart version that changed the default back to what works, but it looks like it’s not this one.

9

so there need to be an update of the prerquisites helm charts?

10

Yeah, you may need to install the schema registry which will be used.

11

attachment
attachment725×176 15.3 KB

12

this is what I currently have in my values.yaml

13

This looks good to me

14

It is with this config that I get the error describes :confused:

15

Did you make the change in values.yaml as well?

16

My bad, I meant this is the current content of my values.yaml file

17

So it is currently failing with these values, so what do you suggest to replace them with?

18

I see I see. Actually, if GMS is coming up, then I think you have that part configured correctly. <@U04N9PYJBEW> can you weigh in on the ingestion failures?

19

:pray:

20

Are you running ingestion from the UI? And are you sure datahub-datahub-gms:8080 is the correct url from inside your cluster? You can try curl datahub-datahub-gms:8080/health to check