Troubleshooting Datahub Deployment on Kubernetes with Custom Images and Helm Charts

Original Slack Thread

Datahub version : 0.10.5
Deployment to : Kubernetes

Hello everyone ! I hope you are having a great day ^^.

I have a bit of trouble to deploy a version of Datahub on Kubernetes using the helm charts with custom GMS and Frontend images.

I am familiar with the process of replacing the images of the GMS and FrontEnd using the values.yaml of the Datahub’s helm chart.
And I also uncommented the part needed to have the schema registry up as it seems to not work without it.

But a have some issues with the GMS, there are a lot of very strange errors that I do not comprehend :sweat_smile:.

(I already search to see if someone had the same issue before me on this Slack community but I did not find anything.)

Attached to this message
• the 2 logs files of the GMS and the upgrade-job (both are failing, but I think the upgrade-job fail because of the GMS failing)
• the 2 values.yaml that I use as well (“values-pre.yaml” if for the prerequisites chart and the “values.yaml” if for Datahub’s chart).

Ps : Everything work when I deploy using docker with the dev docker-compose file according to the documentation, but not on Kubernetes…
Ps 2 : I used to deploy a previous version the 0.8.45 on Kubernetes and everything worked fine, but I am trying to use (a clean/empty instance of) the 0.10.5 right now.attachmentattachmentattachmentattachment

<@U03MF8MU5P0> might be able to speak to this!

Please share the logs from the system-update job. This job is run before other components but after the various setup jobs (*-setup). Containers are listed https://datahubproject.io/docs/architecture/docker-containers|here. GMS is waiting for this job to complete successfully

2023-09-25 12:26:36,695 [ThreadPoolTaskExecutor-1] WARN  c.l.m.b.k.DataHubUpgradeKafkaListener:99 - System version is not up to date: X. Waiting for datahub-upgrade to complete...```

(Thank you for taking the time to view my case ^^.)

Sry I didn’t shared this job’s logs as I only shared the failing containers :sweat_smile:.

Please find it attached to this message ^^.attachment

You need to run the same container versions for all components. The system-update job you have version v0.11.0
2023-09-27 12:39:39,206 [main] INFO c.l.d.u.s.e.steps.DataHubStartupStep:38 - Initiating startup for version: v0.11.0-1
and per the GMS log above v0.10.5
2023-09-25 12:26:36,695 [ThreadPoolTaskExecutor-1] INFO c.l.m.b.k.DataHubUpgradeKafkaListener:92 - Latest system update version: v0.10.5-1
Did you update GMS as well since you posted the last log?