Fixing Datahub Upgrade Errors from 0.8.45 to 0.10.4

all-things-deployment
1

Original Slack Thread

Hi Team
we are upgrading Datahub using helm from 0.8.45 to 0.10.4, we updated helm-chart to 0.2.181…but we are getting below error for mae-consumer and mce-consumer. Could anyone please help us?
MCE error
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘restliServletRegistration’ defined in class path
resource [com/linkedin/metadata/restli/RestliServletConfig.class]: Unsatisfied dependency expressed through method ‘restliServletRegistration’
parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name
‘restliHandlerServlet’: Unsatisfied dependency expressed through field ‘_r2Servlet’; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘rapServlet’
defined in class path resource [com/linkedin/restli/server/RAPServletFactory.class]: Bean instantiation via factory method failed;
nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.r2.transport.http.server.RAPServlet]:
Factory method ‘rapServlet’ threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating
bean with name ‘authorizerChainFactory’: Unsatisfied dependency expressed through field ‘dataHubAuthorizer’; nested exception is
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘dataHubAuthorizerFactory’: Unsatisfied dependency
expressed through field ‘entityClient’; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean
with name ‘javaEntityClientFactory’: Unsatisfied dependency expressed through field ‘_entityService’; nested exception is
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘dataHubKafkaEventProducerFactory’: Unsatisfied
dependency expressed through field ‘kafkaProducer’; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name ‘dataHubKafkaProducerFactory’: Unsatisfied dependency expressed through field ‘_schemaRegistryConfig’; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘awsGlueSchemaRegistryFactory’: Injection of autowired
dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder ‘kafka.schemaRegistry.awsGlue.region’ in
value “${kafka.schemaRegistry.awsGlue.region}”

MAE error:
Error starting ApplicationContext. To display the conditions report re-run your application with ‘debug’ enabled.
2023-08-22 11:15:32,623 [main] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘dataHubKafkaEventProducerFactory’: Unsatisfied
dependency expressed through field ‘kafkaProducer’; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name ‘dataHubKafkaProducerFactory’: Unsatisfied dependency expressed through field ‘_schemaRegistryConfig’; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘awsGlueSchemaRegistryFactory’: Injection of autowired
dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder ‘kafka.schemaRegistry.awsGlue.region’ in
value “${kafka.schemaRegistry.awsGlue.region}”

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘dataHubKafkaProducerFactory’:
Unsatisfied dependency expressed through field ‘_schemaRegistryConfig’; nested exception is org.springframework.beans.factory.BeanCreationException:
Error creating bean with name ‘awsGlueSchemaRegistryFactory’: Injection of autowired dependencies failed; nested exception is
java.lang.IllegalArgumentException: Could not resolve placeholder ‘kafka.schemaRegistry.awsGlue.region’ in value
“${kafka.schemaRegistry.awsGlue.region}”

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘awsGlueSchemaRegistryFactory’: Injection of
autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder
‘kafka.schemaRegistry.awsGlue.region’ in value “${kafka.schemaRegistry.awsGlue.region}”

2

AWS Glue support got broken in v0.10.3 from the Internal Schema Registry changes, there was a recent bug fix for it that will be in the next release

3

https://datahubspace.slack.com/archives/CV2UVAPPG/p1692712076260069?thread_ts=1692278819.950959&cid=CV2UVAPPG

4

Thanks Ryan it worked,
Now I am facing issue with gms pod below are the errors

INFO c.l.g.f.k.s.DUHESchemaRegistryFactory:29 - DataHub System Update Registry
2023-08-24 12:08:41,767 [main] INFO c.l.g.f.k.KafkaEventConsumerFactory:116 - Event-based DUHE KafkaListenerContainerFactory built successfully. Consumer concurrency = 1
2023-08-24 12:08:44,003 [main] ERROR i.c.k.s.client.rest.RestService:267 - Failed to send HTTP request to endpoint: http://schema-registry:port/subjects/MetadataChangeLog_Versioned_v1-value/versions
java.net.UnknownHostException: schema-registry

2023-08-24 12:09:13,589 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-24 12:09:14,595 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve
2023-08-24 12:09:14,595 [pool-13-thread-1] ERROR c.l.m.boot.OnBootApplicationListener:73 - Failed to bootstrap DataHub, OpenAPI servlet was not ready after 30 seconds
2023-08-24 12:09:14.601:INFO:oejs.AbstractConnector:JettyShutdownThread: Stopped ServerConnector@4387b79e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2023-08-24 12:09:14.601:INFO:oejs.session:JettyShutdownThread: node0 Stopped scavenging
2023-08-24 12:09:14.606:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘schemaRegistryServlet’
2023-08-24 12:09:14.613:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘openapiServlet’
2023-08-24 12:09:14.619:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘healthServlet’
2023-08-24 12:09:14.621:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘authServlet’
2023-08-24 12:09:14.621:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘apiServlet’
2023-08-24 12:09:14.626:INFO:oejshC.ROOT:JettyShutdownThread: Closing Spring root WebApplicationContext
2023-08-24 12:09:14.979:INFO:oejsh.ContextHandler:JettyShutdownThread: Stopped o.e.j.w.WebAppContext@6eda5c9{Open source GMS,/,null,STOPPED}{file:///datahub/datahub-gms/bin/war.war}
2023/08/24 12:09:15 Command exited with error: exit status 1

Could anyone please suggest?

5

Yeah this is the second part of the bug in that thread: https://github.com/datahub-project/datahub/pull/8693 I have a fix slated for the next release

6

Since glue doesn’t give you a schema registry url you can put in the configuration, that check fails. The way to work around it would be to temporarily deploy the prerequisite Confluent Schema registry or there may be another url you could pass in there that will return one of the accepted status codes

7

Thanks again Ryan,OnBootApplicationListener.java schema registry issues is resolved
Now getting issues with Elasticsearch index

023-08-24 16:57:46.370:INFO:oejshC.ROOT:main: Initializing Spring DispatcherServlet ‘authServlet’
2023-08-24 16:57:46,370 [main] INFO o.s.web.servlet.DispatcherServlet:525 - Initializing Servlet ‘authServlet’
2023-08-24 16:57:46,429 [pool-16-thread-1] ERROR c.l.g.factory.telemetry.DailyReport:107 - Unable to send telemetry metrics, MixPanel API not initialized

2023-08-24 16:57:52,014 [pool-10-thread-1] ERROR c.l.m.s.e.query.ESSearchDAO:98 - Search query failed
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [url:443], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“query_shard_exception”,“reason”:“[simple_query_string] analyzer [query_word_delimited] not found”,“index_uuid”:“”,“index”:“datahubpolicyindex_v2”}],“type”:“search_phase_execution_exception”,“reason”:“all shards failed”,“phase”:“query”,“grouped”:true,“failed_shards”:[{“shard”:0,“index”:“datahubpolicyindex_v2”,“node”:“68p-15EqQDKradMUiqt-9g”,“reason”:{“type”:“query_shard_exception”,“reason”:“[simple_query_string] analyzer [query_word_delimited] not found”,“index_uuid”:“”,“index”:“datahubpolicyindex_v2”}}]},“status”:400}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
… 17 common frames omitted
2023-08-24 16:57:52,016 [pool-10-thread-1] ERROR c.d.authorization.DataHubAuthorizer:230 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30
com.datahub.util.exception.ESQueryException: Search query failed:
at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:99)
at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:202)
at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:123)
at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:113)

8

subchart pods running fine except for gms…Please assist me.

9

Is this the full log? It just gets to here and GMS does not start up? Sometimes there are issues with updating the policy cache while GMS starts up so it’s unclear if this is blocking. How are you running ElasticSearch? Docker?

10

running using helm chart, we have elasticsearch on AWS

11

Hi <@UV5UEC3LN>
I couldn’t share full logs
2023-08-28 17:29:15,879 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:15,879 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:16,880 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:16,881 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:17,638 [pool-10-thread-1] ERROR c.l.m.s.e.query.ESSearchDAO:98 - Search query failed
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088)
at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:92)
at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:202)
at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:123)
at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:113)
at com.linkedin.metadata.client.JavaEntityClient.search(JavaEntityClient.java:304)
at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:51)
at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:43)
at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:223)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://abc.us-east-1.es.amazonaws.com:443], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{“error”:{“root_cause”:[{“type”:“index_not_found_exception”,“reason”:“no such index [datahubpolicyindex_v2]”,“resource.type”:“index_or_alias”,“resource.id”:“datahubpolicyindex_v2”,“index_uuid”:“na”,“index”:“datahubpolicyindex_v2”}],“type”:“index_not_found_exception”,“reason”:“no such index [datahubpolicyindex_v2]”,“resource.type”:“index_or_alias”,“resource.id”:“datahubpolicyindex_v2”,“index_uuid”:“na”,“index”:“datahubpolicyindex_v2”},“status”:404}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
… 17 common frames omitted
2023-08-28 17:29:17,639 [pool-10-thread-1] ERROR c.d.authorization.DataHubAuthorizer:230 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30
com.datahub.util.exception.ESQueryException: Search query failed:
at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:99)
at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:202)
at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:123)
at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:113)
at

12

com.linkedin.metadata.client.JavaEntityClient.search(JavaEntityClient.java:304)
at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:51)
at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:43)
at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:223)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088)
at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:92)
… 13 common frames omitted
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://abc.us-east-1.es.amazonaws.com:443], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{“error”:{“root_cause”:[{“type”:“index_not_found_exception”,“reason”:“no such index [datahubpolicyindex_v2]”,“resource.type”:“index_or_alias”,“resource.id”:“datahubpolicyindex_v2”,“index_uuid”:“na”,“index”:“datahubpolicyindex_v2”}],“type”:“index_not_found_exception”,“reason”:“no such index [datahubpolicyindex_v2]”,“resource.type”:“index_or_alias”,“resource.id”:“datahubpolicyindex_v2”,“index_uuid”:“na”,“index”:“datahubpolicyindex_v2”},“status”:404}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
… 17 common frames omitted

2023-08-28 17:29:17,882 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:17,882 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:18,883 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:18,884 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:19,885 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:19,885 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:20,891 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve
2023-08-28 17:29:20,891 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:21,892 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:21,892 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:22,893 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:22,893 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:23,894 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:23,894 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:24,895 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:24,896 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:25,896 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:25,897 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:26,898 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:26,898 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:27,899 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:27,899 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:28,900 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:28,901 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:29,902 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:29,902 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:30,908 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve
2023-08-28 17:29:30,909 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:31,910 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:31,910 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:32,911 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:32,911 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:33,912 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:33,912 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:34,913 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:34,914 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:35,914 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:35,915 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:36,916 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:36,916 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:37,917 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:37,917 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:38,918 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:38,919 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:39,920 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry
2023-08-28 17:29:39,920 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second
2023-08-28 17:29:40,927 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve
2023-08-28 17:29:40,928 [pool-13-thread-1] ERROR c.l.m.boot.OnBootApplicationListener:73 - Failed to bootstrap DataHub, OpenAPI servlet was not ready after 30 seconds
2023-08-28 17:29:40.942:INFO:oejs.AbstractConnector:JettyShutdownThread: Stopped ServerConnector@4387b79e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2023-08-28 17:29:40.943:INFO:oejs.session:JettyShutdownThread: node0 Stopped scavenging
2023-08-28 17:29:40.945:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘schemaRegistryServlet’
2023-08-28 17:29:40.948:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘openapiServlet’
2023-08-28 17:29:40.956:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘healthServlet’
2023-08-28 17:29:40.957:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘authServlet’
2023-08-28 17:29:40.958:INFO:oejshC.ROOT:JettyShutdownThread: Destroying Spring FrameworkServlet ‘apiServlet’
2023-08-28 17:29:40.962:INFO:oejshC.ROOT:JettyShutdownThread: Closing Spring root WebApplicationContext
2023-08-28 17:29:41.327:INFO:oejsh.ContextHandler:JettyShutdownThread: Stopped o.e.j.w.WebAppContext@6eda5c9{Open source GMS,/,null,STOPPED}{file:///datahub/datahub-gms/bin/war.war}
2023/08/28 17:29:41 Command exited with error: exit status 1

13

So seems like there are two things going on there:

  1. Your datahubpolicy index doesn’t exist, which indicates there may have been some issues with the ElasticSearch initialization steps
  2. It’s still running into the error I mentioned above, you have an invalid schema registry url specified so the OnBootApplicationListener kills GMS after a bit. This will get fixed for Glue based deployments in the next release
14

I have added changes from your PR to my repository in OnBootApplicationListener, but still same error

15

And you’re set to run with Glue rather than Internal?

16

Is any other files also required changes? As we have elsaticsearch, kafka, rds on AWS