Fixing Serialization Error in DataHub Helm Chart: Enabling Schema Registry & Upcoming Chart Version

Original Slack Thread

Hello! I’m following <|Deploying with Kubernetes | DataHub> and run helm install datahub datahub/datahub in my minikube
but getting the following error….

2023-08-09 00:09:49,248 [main] ERROR c.l.d.u.s.e.steps.DataHubStartupStep:40 - DataHubStartupStep failed.
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: No schema registered under subject!
I’m using the default value but do I need to <|enable schema registry here> ? Thank you!

yes, there is currently a bug in the latest version. See for reference

Thank you for your quick reply! created a PR to fix this :bow:
• <Sign in to GitHub · GitHub enable SPRING_KAFKA_PROPERTIES_AUTO_REGISTER_SCHEMAS by LittleWat · Pull Request #358 · acryldata/datahub-helm>

Thanks for creating the report! <@U03MF8MU5P0> Could you look into this?

Updated both the issue and PR with comments. For the issue, can you confirm the values settings, I am guessing KAFKA schema registry was enabled & it was pointing to GMS instead of the confluent registry service?

Thank you for your reply!
This datahub-0.2.181 value file.
It uses INTERNAL one. You should be able to reproduce the error.

just installing datahub-0.2.181 will reproduce the error.

I’ve had time to look at this in more detail. I see what you mean. The env var is indeed no longer needed to be set to false and a hold over from a previous method of mocking the schema registry for the upgrade job. Sorry for the delay.

The next version of the chart is building, 0.2.183 , thank you! :pray:

Thank you for your investigation and merging the PR :bow: