Troubleshooting GMS Connection Errors in Datahub on OCP Kubernetes

troubleshoot
1

Original Slack Thread

Hi Team,
I posted my query on an existing chain, didn’t get any response so posting it here again.

GMS doesn’t start at all for us. It gets connection refused error from OpenServlet and it exits with Bootstrap error. We are trying to host Datahub on OCP using Kubernetes. Datahub version 0.10.4.

GMS container logs:
INFO:oejshC.ROOT:main: Initializing Spring DispatcherServlet 'healthServlet'
[main] INFO o.s.web.servlet.DispatcherServlet:525 - Initializing Servlet 'healthServlet'
[pool-15-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: Connect to localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
[pool-15-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
[pool-15-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: Connect to localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
[pool-15-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
...
[R2 Nio Event Loop-1-1] WARN c.l.r.t.h.c.c.ChannelPoolLifecycle:139 - Failed to create channel, remote=localhost/127.0.0.1:8080
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
...
[pool-15-thread-1] ERROR c.l.m.boot.OnBootApplicationListener:73 - Failed to bootstrap DataHub, OpenAPI servlet was not ready after 30 seconds
2023/11/27 12:53:55 Command exited with error: exit status 1

No-code Migration logs:
[main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - ERROR: Cannot connect to GMSat <http://host> datahub-datahub-gms port 8080. Make sure GMS is on the latest version and is running at that host before starting the migration.
[main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - Failed Step 2/6: GMSQualificationStep. Failed after 2 retries.
2023-11-27 14:10:58,575 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - Exiting upgrade NoCodeDataMigration with failure.
[main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - Upgrade NoCodeDataMigration completed with result FAILED. Exiting...

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

<@U063WTWMH2L> Would love your help here!