Troubleshooting MySQL Setup Job Timeout and GMS Pod Errors

Original Slack Thread

Hi, we are deploying datahub to AWS EKS, and the prerequisites are deployed and healthy mysql in particualar, when I deploy datahub using helm, the datahub-mysql-setup-job-xxxx timeout with below

Error
2023/11/30 15:10:05 Problem with dial: dial tcp [fd99:4da0:d09::9cd9]:3306: connect: connection refused. Sleeping 1s
2023/11/30 15:10:05 Timeout after 4m0s waiting on dependencies to become available: [tcp://prerequisites-mysql:3306]

mysql pod log is ready to accept connections,
2023-11-30T12:23:31.342704Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld: ready for connections. Version: ‘8.0.29’ socket: ‘/opt/bitnami/mysql/tmp/mysql.sock’ port: 3306 Source distribution.

Check if you are using a CNI plugin. If so, make sure it is configured to allow connections between the pods. Make sure the mysql service is also configured for ipv6 since that seems to be what the mysql-setup job is using.

Hi <@U03MF8MU5P0> thanks for the response, yes the cluster has CNI addon enabled and mysql pod is also running on ipv6

we saw the mysql bind address as 0.0.0.0 and could not listen to incoming requests, as we could not make changes to the mysql configuration we decided to goahead with postgres and it worked fine.

gms pod is not coming up and has below errors

2023-12-07 13:37:32,552 [qtp2141817446-18] WARN c.d.a.a.AuthenticatorChain:89 - Authentication chain failed to resolve a valid authentication. Errors: [(com.datahub.authentication.authenticator.DataHubSystemAuthenticator,Failed to authenticate inbound request: Authorization header is missing Authorization header.), (com.datahub.authentication.authenticator.DataHubTokenAuthenticator,Failed to authenticate inbound request: Request is missing ‘Authorization’ header.)]

That’s an odd warning message but unfortunately is not enough information to debug the reason it is not starting. Please share the full gms log