Issue with Kafka Version Mismatch during Datahub Upgrade in Kubernetes Deployment with Helm Chart

getting-started
1

Original Slack Thread

Hello all, We were using the datahub in the Kubernetes deployed through the helm chart. It was working fine.
We re-installed the datahub again through latest helm chart but it seems there is some version issue with the Kafka

datahub-datahub-system-update-job pod logs:

2023-08-31 07:24:47,355 [SpringApplicationShutdownHook] INFO  o.a.k.clients.producer.KafkaProducer:1182 - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2```![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F05PYTWCYRM/image.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
2

cc: <@U05C0HZS3EF>

3

Hi, we are currently trying to upgrade our datahub from v 0.10.4 to 0.10.5 with the helm chart and we are having exactly the same issue. For us it seems there’s to come before the ANTR issue.

4

Stack trace part 1 :

2023-08-31T08:28:29.598752054Z 2023-08-31 08:28:29,598 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Executing Step 3/6: BuildIndicesPostStep...
2023-08-31T08:28:29.599713313Z 2023-08-31 08:28:29,599 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Completed Step 3/6: BuildIndicesPostStep successfully.
2023-08-31T08:28:29.599752546Z 2023-08-31 08:28:29,599 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Executing Step 4/6: DataHubStartupStep...
2023-08-31T08:28:29.797120807Z 2023-08-31 08:28:29,796 [main] ERROR c.l.d.u.s.e.steps.DataHubStartupStep:40 - DataHubStartupStep failed.
2023-08-31T08:28:29.797153248Z org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
2023-08-31T08:28:29.797158951Z Caused by: java.io.IOException: No sche
ma registered under subject!
2023-08-31T08:28:29.797162924Z  at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestVersion(MockSchemaRegistryClient.java:261)
2023-08-31T08:28:29.797166268Z  at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestSchemaMetadata(MockSchemaRegistryClient.java:310)
2023-08-31T08:28:29.797170350Z  at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe.lookupLatestVersion(AbstractKafkaSchemaSerDe.java:181)
2023-08-31T08:28:29.797174366Z  at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:77)
2023-08-31T08:28:29.797177642Z  at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59)
2023-08-31T08:28:29.797180930Z  at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
2023-08-31T08:28:29.797184776Z  at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:902)
2023-08-31T08:28:29.797198819Z  at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)
2023-08-31T08:28:29.797204653Z  at com.linkedin.metadata.dao.producer.KafkaEventProducer.produceDataHubUpgradeHistoryEvent(KafkaEventProducer.java:171)
2023-08-31T08:28:29.797209693Z  at com.linkedin.datahub.upgrade.system.elasticsearch.steps.DataHubStartupStep.lambda$executable$0(DataHubStartupStep.java:37)
2023-08-31T08:28:29.797219533Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
2023-08-31T08:28:29.797224225Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
2023-08-31T08:28:29.797228919Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
2023-08-31T08:28:29.797233931Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
2023-08-31T08:28:29.797239514Z  at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.j
ava:80)
2023-08-31T08:28:29.797244816Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
2023-08-31T08:28:29.797252653Z  at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
2023-08-31T08:28:29.797258933Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
2023-08-31T08:28:29.797262640Z  at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
2023-08-31T08:28:29.797268191Z  at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:23)
2023-08-31T08:28:29.797276535Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-08-31T08:28:29.797285343Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-31T08:28:29.797290533Z  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-31T08:28:29.797296934Z  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2023-08-31T08:28:29.797302277Z  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
2023-08-31T08:28:29.797311348Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
2023-08-31T08:28:29.797316823Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
2023-08-31T08:28:29.797322544Z  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
2023-08-31T08:28:29.797656160Z 2023-08-31 08:28:29,797 [main] ERROR c.l.d.u.s.e.steps.DataHubStartupStep:40 - DataHubStartupStep failed.
2023-08-31T08:28:29.797682484Z org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
2023-08-31T08:28:29.797688883Z Caused by: java.io.IOException: No schema registered under subject!
2023-08-31T08:28:29.797693831Z  at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestVersion(MockSchemaRegistryClient.java:261)
2023-08-31T08:28:29.797698930Z  a
t io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestSchemaMetadata(MockSchemaRegistryClient.java:310)
2023-08-31T08:28:29.797703533Z  at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe.lookupLatestVersion(AbstractKafkaSchemaSerDe.java:181)
2023-08-31T08:28:29.797708127Z  at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:77)
2023-08-31T08:28:29.797713605Z  at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59)
2023-08-31T08:28:29.797724760Z  at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
2023-08-31T08:28:29.797740569Z  at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:902)
2023-08-31T08:28:29.797745009Z  at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)
2023-08-31T08:28:29.797749254Z  at com.linkedin.metadata.dao.producer.KafkaEventProducer.produceDataHubUpgradeHistoryEvent(KafkaEventProducer.java:171)
2023-08-31T08:28:29.797753566Z  at com.linkedin.datahub.upgrade.system.elasticsearch.steps.DataHubStartupStep.lambda$executable$0(DataHubStartupStep.java:37)
2023-08-31T08:28:29.797758051Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
2023-08-31T08:28:29.797762535Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
2023-08-31T08:28:29.797770389Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
2023-08-31T08:28:29.797774976Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
2023-08-31T08:28:29.797779420Z  at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:80)
2023-08-31T08:28:29.797784239Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
2023-08-31T08:28:29.797788787Z  at org.springframework.boot.SpringApplication.callRunners(
SpringApplication.java:752)```
5

Part 2, if it can help to solve this issue

2023-08-31T08:28:29.797797966Z  at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
2023-08-31T08:28:29.797802400Z  at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:23)
2023-08-31T08:28:29.797809822Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-08-31T08:28:29.797816570Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-31T08:28:29.797821077Z  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-31T08:28:29.797825611Z  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2023-08-31T08:28:29.797830622Z  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
2023-08-31T08:28:29.797834981Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
2023-08-31T08:28:29.797839371Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
2023-08-31T08:28:29.797847107Z  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
2023-08-31T08:28:29.798061023Z 2023-08-31 08:28:29,797 [main] ERROR c.l.d.u.s.e.steps.DataHubStartupStep:40 - DataHubStartupStep failed.
2023-08-31T08:28:29.798070631Z org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
2023-08-31T08:28:29.798075139Z Caused by: java.io.IOException: No schema registered under subject!
2023-08-31T08:28:29.798079450Z  at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestVersion(MockSchemaRegistryClient.java:261)
2023-08-31T08:28:29.798083766Z  at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestSchemaMetadata(MockSchemaRegistryClient.java:310)
2023-08-31T08:28:29.798088294Z  at io.confluent.kafka.serializers.AbstractKafkaSchemaSer
De.lookupLatestVersion(AbstractKafkaSchemaSerDe.java:181)
2023-08-31T08:28:29.798092581Z  at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:77)
2023-08-31T08:28:29.798096906Z  at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59)
2023-08-31T08:28:29.798101625Z  at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
2023-08-31T08:28:29.798130021Z  at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:902)
2023-08-31T08:28:29.798134939Z  at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)
2023-08-31T08:28:29.798139378Z  at com.linkedin.metadata.dao.producer.KafkaEventProducer.produceDataHubUpgradeHistoryEvent(KafkaEventProducer.java:171)
2023-08-31T08:28:29.798143583Z  at com.linkedin.datahub.upgrade.system.elasticsearch.steps.DataHubStartupStep.lambda$executable$0(DataHubStartupStep.java:37)
2023-08-31T08:28:29.798148094Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
2023-08-31T08:28:29.798154534Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
2023-08-31T08:28:29.798162340Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
2023-08-31T08:28:29.798166913Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
2023-08-31T08:28:29.798171233Z  at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:80)
2023-08-31T08:28:29.798178157Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
2023-08-31T08:28:29.798182399Z  at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
2023-08-31T08:28:29.798190326Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
2023-08-31T08:28:29.798194985Z  at org.springframework.boot.builder.SpringApp
licationBuilder.run(SpringApplicationBuilder.java:164)
2023-08-31T08:28:29.798203483Z  at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:23)
2023-08-31T08:28:29.798207800Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-08-31T08:28:29.798212155Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-31T08:28:29.798216720Z  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-31T08:28:29.798221002Z  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2023-08-31T08:28:29.798225435Z  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
2023-08-31T08:28:29.798230536Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
2023-08-31T08:28:29.798238585Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
2023-08-31T08:28:29.798242897Z  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
2023-08-31T08:28:29.798516233Z 2023-08-31 08:28:29,798 [main] ERROR c.l.d.u.s.e.steps.DataHubStartupStep:40 - DataHubStartupStep failed.
2023-08-31T08:28:29.798539321Z org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
2023-08-31T08:28:29.798546175Z Caused by: java.io.IOException: No schema registered under subject!
2023-08-31T08:28:29.798551861Z  at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestVersion(MockSchemaRegistryClient.java:261)
2023-08-31T08:28:29.798558074Z  at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestSchemaMetadata(MockSchemaRegistryClient.java:310)
2023-08-31T08:28:29.798563842Z  at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe.lookupLatestVersion(AbstractKafkaSchemaSerDe.java:181)
2023-08-31T08:28:29.798570120Z  at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:77)
2023-08-31T0
8:28:29.798578736Z  at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59)
2023-08-31T08:28:29.798594308Z  at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
2023-08-31T08:28:29.798600209Z  at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:902)
2023-08-31T08:28:29.798605865Z  at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)
2023-08-31T08:28:29.798611816Z  at com.linkedin.metadata.dao.producer.KafkaEventProducer.produceDataHubUpgradeHistoryEvent(KafkaEventProducer.java:171)
2023-08-31T08:28:29.798626660Z  at com.linkedin.datahub.upgrade.system.elasticsearch.steps.DataHubStartupStep.lambda$executable$0(DataHubStartupStep.java:37)
2023-08-31T08:28:29.798633109Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
2023-08-31T08:28:29.798638956Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
2023-08-31T08:28:29.798644255Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
2023-08-31T08:28:29.798648948Z  at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
2023-08-31T08:28:29.798654053Z  at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:80)
2023-08-31T08:28:29.798659064Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
2023-08-31T08:28:29.798663256Z  at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
2023-08-31T08:28:29.798670061Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
2023-08-31T08:28:29.798677109Z  at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
2023-08-31T08:28:29.798682027Z  at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:23)
2023-08-31T08:28:29.798687438Z  at jav
a.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-08-31T08:28:29.798692537Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-08-31T08:28:29.798697306Z  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-08-31T08:28:29.798702253Z  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2023-08-31T08:28:29.798706902Z  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
2023-08-31T08:28:29.798711613Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
2023-08-31T08:28:29.798716551Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
2023-08-31T08:28:29.798723269Z  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
2023-08-31T08:28:29.798738505Z 2023-08-31 08:28:29,798 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Failed Step 4/6: DataHubStartupStep. Failed after 3 retries.
2023-08-31T08:28:29.798762986Z 2023-08-31 08:28:29,798 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Exiting upgrade SystemUpdate with failure.
2023-08-31T08:28:29.799215552Z 2023-08-31 08:28:29,799 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Upgrade SystemUpdate completed with result FAILED. Exiting...```
6

<@U05QDARQ2UB> we have solved this issue using this thread : https://datahubspace.slack.com/archives/C029A3M079U/p1692956290598499

7

<@U05QDARQ2UB>