Hi Everyone, I am trying to deploy datahub 0.12 version on our kubernets cluster . Though all prerequisites looks good and elastic search endpoint is UP and running and also the elastic search setup Job is exited with 200 code. Still GMS is crashing with below error. Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088)
at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:260)
… 14 common frames omitted
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch-master:9200], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
Warnings: [Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security., [ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices.]
{“error”:{“root_cause”:[{“type”:“index_not_found_exception”,“reason”:“no such index [datahub_usage_event]“,”resource.type”:“index_or_alias”,“resource.id”:“datahub_usage_event”,“index_uuid”:“na“,”index”:“datahub_usage_event”}],“type”:“index_not_found_exception”,“reason”:“no such index [datahub_usage_event]“,”resource.type”:“index_or_alias”,“resource.id”:“datahub_usage_event”,“index_uuid”:“na“,”index”:“datahub_usage_event”},“status”:404}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
… 18 common frames omitted
can someone guide us with thi please.
$ kubectl get all -n development
NAME READY STATUS RESTARTS AGE
pod/datahub-acryl-datahub-actions-54b6d7fd49-4gmb4 1/1 Running 5 (2m56s ago) 24m
pod/datahub-datahub-frontend-54665df759-lp6sw 1/1 Running 0 24m
pod/datahub-datahub-gms-6546684dc8-nmq6f 0/1 Running 4 (4m21s ago) 24m
pod/datahub-datahub-system-update-job-dz5hn 0/1 Completed 0 26m
pod/datahub-elasticsearch-setup-job-pkc89 0/1 Completed 0 27m
pod/datahub-kafka-setup-job-dv6gs 0/1 Completed 0 27m
pod/datahub-mysql-setup-job-hmfll 0/1 Completed 0 26m
pod/datahub-nocode-migration-job-478d8 0/1 Error 0 22m
pod/datahub-nocode-migration-job-hvpd5 0/1 Error 0 5m6s
pod/datahub-nocode-migration-job-jslpl 0/1 Error 0 24m
pod/datahub-nocode-migration-job-shh2g 0/1 Error 0 10m
pod/datahub-nocode-migration-job-wnvdf 0/1 Error 0 14m
pod/datahub-nocode-migration-job-x55l4 0/1 Error 0 17m
pod/datahub-nocode-migration-job-zlwlq 0/1 Error 0 19m
pod/elasticsearch-master-0 1/1 Running 0 21h
pod/prerequisites-hfgqy-test 0/1 Completed 0 20m
pod/prerequisites-kafka-0 1/1 Running 0 21h
pod/prerequisites-mysql-0 1/1 Running 0 21h
pod/prerequisites-zookeeper-0 1/1 Running 0 21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/datahub-acryl-datahub-actions ClusterIP 192.168.7.187 <none> 9093/TCP 24m
service/datahub-datahub-frontend LoadBalancer 192.168.1.198 <pending> 9002:9031/TCP,4318:9337/TCP 24m
service/datahub-datahub-gms LoadBalancer 192.168.3.235 <pending> 8080:9803/TCP,4318:9926/TCP 24m
service/elasticsearch-master ClusterIP 192.168.5.70 <none> 9200/TCP,9300/TCP 21h
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 21h
service/prerequisites-kafka ClusterIP 192.168.5.15 <none> 9092/TCP 21h
service/prerequisites-kafka-headless ClusterIP None <none> 9092/TCP,9094/TCP 21h
service/prerequisites-mysql ClusterIP 192.168.1.85 <none> 3306/TCP 21h
service/prerequisites-mysql-headless ClusterIP None <none> 3306/TCP 21h
service/prerequisites-zookeeper ClusterIP 192.168.1.141 <none> 2181/TCP,2888/TCP,3888/TCP 21h
service/prerequisites-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/datahub-acryl-datahub-actions 1/1 1 1 24m
deployment.apps/datahub-datahub-frontend 1/1 1 1 24m
deployment.apps/datahub-datahub-gms 0/1 1 0 24m
NAME DESIRED CURRENT READY AGE
replicaset.apps/datahub-acryl-datahub-actions-54b6d7fd49 1 1 1 24m
replicaset.apps/datahub-datahub-frontend-54665df759 1 1 1 24m
replicaset.apps/datahub-datahub-gms-6546684dc8 1 1 0 24m
NAME READY AGE
statefulset.apps/elasticsearch-master 1/1 21h
statefulset.apps/prerequisites-kafka 1/1 21h
statefulset.apps/prerequisites-mysql 1/1 21h
statefulset.apps/prerequisites-zookeeper 1/1 21h
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/datahub-datahub-cleanup-job-template * * * * * True 0 <none> 24m
cronjob.batch/datahub-datahub-restore-indices-job-template * * * * * True 0 <none> 24m
NAME COMPLETIONS DURATION AGE
job.batch/datahub-datahub-system-update-job 1/1 2m24s 26m
job.batch/datahub-elasticsearch-setup-job 1/1 7s 27m
job.batch/datahub-kafka-setup-job 1/1 51s 27m
job.batch/datahub-mysql-setup-job 1/1 7s 26m
job.batch/datahub-nocode-migration-job 0/1 24m 24m
Hi Siva! I just came across this one and noticed there wasn’t a resolution… if you’re still hitting this issue, I highly suggest joining us at the Community Marathon tomorrow! We have a variety of troubleshooting & support sessions scheduled throughout the day where we can give live help 
https://datahubspace.slack.com/archives/CUMV92XRQ/p1705433341623299