Troubleshooting datahub-upgrade SystemUpdate Exception and Java 17 Compatibility

all-things-deployment
1

Original Slack Thread

Hi,

I am running datahub-upgrade with SystemUpdate parameter and getting below exception. It says index not found. Ideally the index should get created if it doesn’t exist

	at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:209)
	at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)
	at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)
	at org.opensearch.client.IndicesClient.delete(IndicesClient.java:118)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.reindex(ESIndexBuilder.java:358)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.buildIndex(ESIndexBuilder.java:225)
	... 22 common frames omitted
	Suppressed: org.opensearch.client.ResponseException: method [DELETE], host [<https://c9c1211d1a9049b0b64be0193936cdb7.ecenp.visa.com:9243>], URI [/vdc_dev_containerindex_v2_1702876349208?master_timeout=30s&amp;ignore_unavailable=false&amp;expand_wildcards=open%2Cclosed&amp;allow_no_indices=true&amp;ignore_throttled=false&amp;timeout=30s], status line [HTTP/1.1 404 Not Found]
Warnings: [[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices.]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [vdc_dev_containerindex_v2_1702876349208]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"vdc_dev_containerindex_v2_1702876349208","index":"vdc_dev_containerindex_v2_1702876349208"}],"type":"index_not_found_exception","reason":"no such index [vdc_dev_containerindex_v2_1702876349208]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"vdc_dev_containerindex_v2_1702876349208","index":"vdc_dev_containerindex_v2_1702876349208"},"status":404}
		at org.opensearch.client.RestClient.convertResponse(RestClient.java:375)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:345)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
		at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)
		... 27 common frames omitted```
2

Could there be another process running modifying your index state simultaneously? We do have a check that runs immediately before this code executes that verifies the index exists, so not sure how it would reach this path with the index not existing. See https://github.com/datahub-project/datahub/blob/master/metadata-io/src/main/java/com/linkedin/metadata/search/elasticsearch/indexbuilder/ESIndexBuilder.java#L198|here. Also what version are you running?

3

<@UV5UEC3LN>
I am using Datahub v0.12.0. and Java 17
When I tried using Java 11 it worked on my laptop. But all our Environment nodes are on Java 17.
No other process is running or modifying the index. Yesterday, I checked the code I also came to the same conclusion that this path would not even come.
Could there be a possibility that OpenSearch doesn’t work as expected on Java 17?

4

It’s hard to know, we haven’t done any testing on Java 17 with the current release. ~We are working on getting the project up to Java 17 currently though~. <https://github.com/datahub-project/datahub/pull/9458|(PR got merged)> If it’s possible definitely at least try with Java 11.

5

For us Java 11 is not possible on our environments due to cybersecurity concerns. But I will try to do a fresh setup on a different node with Java 17 to be 100% sure something else is not causing this issue.