Error Feeding Bulk Request: Issues with Elasticsearch and Dataset Visibility

all-things-deployment
1

Original Slack Thread

Hi team, I am getting below error in gms logs and also not able to see datasets only platform is added .
Logs are here…

ERROR c.l.m.s.e.update.BulkListener:56 - Error feeding bulk request. No retries left. Request: Failed to perform bulk request: index [dat
asetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGM%2CPROD%29];Failed to perform bulk request: index [
datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGMGRP%2CPROD%29];Failed to perform bulk request: i
ndex [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGMGRP%2CPROD%29];Failed to perform bulk requ
est: index [system_metadata_service_v1], optype: [UPDATE], type [_doc], id [MulrbF+m7pOX8FZWKhFvfA==];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], typ
e [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTG%2CPROD%29];Failed to perform bulk request: index [system_metadata_service_v1], optype:
[UPDATE], type [_doc], id [+dVcgKkZo0b54/VULBLKVA==];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3A
dataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGL%2CPROD%29];Failed to perform bulk request: index [graph_service_v1], optype: [UPDATE], type [_doc], id [q8v+RyJ/4GuJiUEeie9BYQ==];Faile
d to perform bulk request: index [system_metadata_service_v1], optype: [UPDATE], type [_doc], id [n3GQtcgTWWWXYCsCtIxcIA==];Failed to perform bulk request: index [datasetindex_v2],
optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGL%2CPROD%29];Failed to perform bulk request: index [system_metadat
a_service_v1], optype: [UPDATE], type [_doc], id [1Pe69BbgsIXutJ+qQx46Sg==];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Ad
ataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGL%2CPROD%29];Failed to perform bulk request: index [system_metadata_service_v1], optype: [UPDATE], type [_doc], id
[0EvAuIBMmx2PfhUA010aFQ==];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CC
TGS.STNET.DTS2.CTGL%2CPROD%29];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%
2CCTGS.STNET.DTS2.CTGRP%2CPROD%29];Failed to perform bulk request: index [system_metadata_service_v1], optype: [UPDATE], type [_doc], id [yvOt1wkWazchptZYmJtRzg==];Failed to perfor
m bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGL%2CPROD%29];Failed to per
form bulk request: index [system_metadata_service_v1], optype: [UPDATE], type [_doc], id [OvHgSEtTPUICetikdlM0+g==];Failed to perform bulk request: index [datasetindex_v2], optype:
[UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGL%2CPROD%29];Failed to perform bulk request: index [system_metadata_servic
e_v1], optype: [UPDATE], type [_doc], id [7c9EvHBwA7Xss4O7rNsE2g==];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3
A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGL%2CPROD%29];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adatase
t%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGRP%2CPROD%29];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Ada
taset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTITLE%2CPROD%29];Failed to perform bulk request: index [system_metadata_service_v1], optype: [UPDATE], type [_doc], i
d [sQtpzVgs7goDUM/HXNvkGw==];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2C
CTGS.STNET.DTS2.CTGL%2CPROD%29];Failed to perform bulk request: index [system_metadata_service_v1], optype: [UPDATE], type [_doc], id [qIM8ghhj7xP2qzARnIDXwQ==];Failed to perform b
ulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGM%2CPROD%29];Failed to perfor
m bulk request: index [graph_service_v1], optype: [UPDATE], type [_doc], id [q7vdPq8m0+zZpY5saCJV6A==];Failed to perform bulk request: index [system_metadata_service_v1], optype: [
UPDATE], type [_doc], id [6OV/6/4OgKUF/Y5Rp0hbEg==];Failed to perform bulk request: index [datasetindex_v2], optype: [UPDATE], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3Ad
ataPlatform%3AAmpere%2CCTGS.STNET.DTS2.CTGM%2CPROD%29]
java.io.IOException: Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://logging-es-http.monitoring.svc.cluster.local:9200|http://logging-es-http.monitoring.svc.cluster.local:9200, response
=HTTP/1.1 200 OK}
at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:1783)
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:636)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:376)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:370)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException: null
at java.base/java.util.Objects.requireNonNull(Objects.java:221)
at org.elasticsearch.action.DocWriteResponse.<init>(DocWriteResponse.java:127)
at org.elasticsearch.action.update.UpdateResponse.<init>(UpdateResponse.java:65)
at org.elasticsearch.action.update.UpdateResponse$Builder.build(UpdateResponse.java:172)
at org.elasticsearch.action.update.UpdateResponse$Builder.build(UpdateResponse.java:160)
at org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:159)
at org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:188)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$10(RestHighLevelClient.java:1699)
at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:1781)
… 18 common frames omitted
2023-09-07 14:55:08,612 [qtp71399214-18] INFO c.l.m.r.platform.PlatformResource:61 - Emitting platform event. name: entityChangeEvent, key: entityChangeEvent-urn:li:dataset:(urn:l
i:dataPlatform:Ampere,CTGS.STNET.DTS2.CTGM,PROD)

Any idea on why it is failing while making bulk request to elastic search?

Context : using helm deployment of datahub and ingesting a custom data source.

2

The error indicates it cannot parse the response from Elasticsearch, would you be able to check the Elasticsearch logs? Is there any indication that Elasticsearch is encountering an issue?

3

Which version of Elasticsearch are you running. Perhaps using opensearch 2.x?

4

Elastic search version is 8.2.3

5

Please downgrade to 7.x

6

hi <@U03MF8MU5P0>
I encountered similar issue as mentioned by <@U0550CM33LG>.
May I know whether the build from this https://github.com/datahub-project/datahub/pull/8852|PR is available for use ? I would like to give it a try.

the below log in gms pod is recurring when I refresh the frontend landing page.
│ 2023-09-28 04:04:35,395 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:56 - Error feeding bulk request. No retries left. Request: Failed to perform bulk request: index [datahub_usage_event], optype: [CREATE], type [_doc], id [Recommendation │ java.io.IOException: Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=https://&lt;aws-opensearch-endpoint&gt;:443, response=HTTP/1.1 200 OK} │
besides , The already ingested entities, like domain , tags, and even user groups have not been shown up properly as expected. I wonder whether it also has to do with version conflict.

info:
OpenSearch 2.5
helm chart used : datahub/datahub (0.2.188)

thanks

7

While the PR has been merged to master, we’ve not yet collected and tested all the changes for the next release. Assuming the next release version will likely be v0.11.1 and will include the OS 2.x support in the changelog for that release.