Troubleshooting datahub-upgrade SystemUpdate Exception and Java 17 Compatibility

Original Slack Thread


I am running datahub-upgrade with SystemUpdate parameter and getting below exception. It says index not found. Ideally the index should get created if it doesn’t exist

	at org.opensearch.client.RestHighLevelClient.parseEntity(
	at org.opensearch.client.RestHighLevelClient.parseResponseException(
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(
	at org.opensearch.client.RestHighLevelClient.performRequest(
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(
	at org.opensearch.client.IndicesClient.delete(
	... 22 common frames omitted
	Suppressed: org.opensearch.client.ResponseException: method [DELETE], host [<>], URI [/vdc_dev_containerindex_v2_1702876349208?master_timeout=30s&amp;ignore_unavailable=false&amp;expand_wildcards=open%2Cclosed&amp;allow_no_indices=true&amp;ignore_throttled=false&amp;timeout=30s], status line [HTTP/1.1 404 Not Found]
Warnings: [[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices.]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [vdc_dev_containerindex_v2_1702876349208]","index_uuid":"_na_","resource.type":"index_or_alias","":"vdc_dev_containerindex_v2_1702876349208","index":"vdc_dev_containerindex_v2_1702876349208"}],"type":"index_not_found_exception","reason":"no such index [vdc_dev_containerindex_v2_1702876349208]","index_uuid":"_na_","resource.type":"index_or_alias","":"vdc_dev_containerindex_v2_1702876349208","index":"vdc_dev_containerindex_v2_1702876349208"},"status":404}
		at org.opensearch.client.RestClient.convertResponse(
		at org.opensearch.client.RestClient.performRequest(
		at org.opensearch.client.RestClient.performRequest(
		at org.opensearch.client.RestHighLevelClient.internalPerformRequest(
		... 27 common frames omitted```

Could there be another process running modifying your index state simultaneously? We do have a check that runs immediately before this code executes that verifies the index exists, so not sure how it would reach this path with the index not existing. See|here. Also what version are you running?

I am using Datahub v0.12.0. and Java 17
When I tried using Java 11 it worked on my laptop. But all our Environment nodes are on Java 17.
No other process is running or modifying the index. Yesterday, I checked the code I also came to the same conclusion that this path would not even come.
Could there be a possibility that OpenSearch doesn’t work as expected on Java 17?

It’s hard to know, we haven’t done any testing on Java 17 with the current release. ~We are working on getting the project up to Java 17 currently though~. <|(PR got merged)> If it’s possible definitely at least try with Java 11.

For us Java 11 is not possible on our environments due to cybersecurity concerns. But I will try to do a fresh setup on a different node with Java 17 to be 100% sure something else is not causing this issue.