Troubleshooting issues with datahub-gms pod and datahub upgrade

all-things-deployment
1

Original Slack Thread

Hi team,
I am facing issue with datahub-gms pod below are the errors:
INFO c.l.g.f.k.s.DUHESchemaRegistryFactory:29 - DataHub System Update Registry
2023-08-24 12:08:41,767 [main] INFO c.l.g.f.k.KafkaEventConsumerFactory:116 - Event-based DUHE KafkaListenerContainerFactory built successfully. Consumer concurrency = 1
2023-08-24 12:08:44,003 [main] ERROR i.c.k.s.client.rest.RestService:267 - Failed to send HTTP request to endpoint: http://schema-registry:8081/subjects/MetadataChangeLog_Versioned_v1-value/versions
java.net.UnknownHostException: schema-registry

2023-08-24 12:09:13,589 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-24 12:09:14,595 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve
2023-08-24 12:09:14,595 [pool-13-thread-1] ERROR c.l.m.boot.OnBootApplicationListener:73 - Failed to bootstrap DataHub, OpenAPI servlet was not ready after 30 seconds
2023-08-24 12:09:14.601:INFO:oejs.AbstractConnector:JettyShutdownThread: Stopped ServerConnector@4387b79e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2023-08-24 12:09:14.601:INFO:oejs.session:JettyShutdownThread: node0 Stopped scavenging
2023-08-24 12:09:14.606:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘schemaRegistryServlet’
2023-08-24 12:09:14.613:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘openapiServlet’
2023-08-24 12:09:14.619:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘healthServlet’
2023-08-24 12:09:14.621:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘authServlet’
2023-08-24 12:09:14.621:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘apiServlet’
2023-08-24 12:09:14.626:INFO:oejshC.ROOT:JettyShutdownThread: Closing Spring root WebApplicationContext
2023-08-24 12:09:14.979:INFO:oejsh.ContextHandler:JettyShutdownThread: Stopped o.e.j.w.WebAppContext@6eda5c9{Open source GMS,/,null,STOPPED}{file:///datahub/datahub-gms/bin/war.war}
2023/08/24 12:09:15 Command exited with error: exit status 1

Could anyone please help?

2

Earlier we were getting above issue to resolve that we try to stop datahub and run datahub-upgrade now we are getting:

2023-08-24 14:43:47,493 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - ERROR: Cannot connect to GMSat http://host datahub-datahub-gms port 8080. Make sure GMS is on the latest version and is running at that host before starting the migration.
2023-08-24 14:43:47,494 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - Failed Step 2/6: GMSQualificationStep. Failed after 2 retries.
2023-08-24 14:43:47,494 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - Exiting upgrade NoCodeDataMigration with failure.
2023-08-24 14:43:47,495 [pool-11-thread-1] ERROR c.l.m.s.e.query.ESSearchDAO:98 - Search query failed
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]

3

Closing the loop here because this has been followed up on here. :slightly_smiling_face: : https://datahubspace.slack.com/archives/CV2UVAPPG/p1692879362997919?thread_ts=1692720924.016559&cid=CV2UVAPPG closing