Hello folks just wondering if anybody has faced with such error on the metadataChangeProposel kafka event handling ?
2024-01-09 13:19:34 2024-01-09 12:19:34,451 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeProposalsProcessor:76 - Error while processing FMCP: FailedMetadataChangeProposal - {error=com.linkedin.r2.RemoteInvocationException: com.linkedin.data.template.RequiredFieldNotPresentException: Field "value" is required but it is not present
I researched this issue a bit and found that any error that occurs on the restli server endpoint causes this error on the client side. It happens because for some reason resli framework doesn’t set the necessary header for an error thrown from the endpoint, so the resli client acts like the response is successful and tries to get a response body that doesn’t exist.
As a result, any error in handling the metadataChangeProposal event causes the retry logic to not occur because the RequiredFieldNotPresentException exception is hard-coded as a skip, and the datahub always sends the same message to the metadataChangeProposalFailed topic, so there is no way to react to a specific error.
Hey there! 
Make sure your message includes the following information if relevant, so we can help more effectively!
- Are you using UI or CLI for ingestion?
- Which DataHub version are you using? (e.g. 0.12.0)
- What data source(s) are you integrating with DataHub? (e.g. BigQuery)
2024-01-09 13:19:34 at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
2024-01-09 13:19:34 at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponseImpl(ResponseFutureImpl.java:130)
2024-01-09 13:19:34 at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponse(ResponseFutureImpl.java:94)
2024-01-09 13:19:34 at com.linkedin.common.client.BaseClient.sendClientRequest(BaseClient.java:55)
2024-01-09 13:19:34 at com.linkedin.entity.client.RestliEntityClient.ingestProposal(RestliEntityClient.java:797)
2024-01-09 13:19:34 at com.linkedin.entity.client.SystemEntityClient.ingestProposal(SystemEntityClient.java:85)
2024-01-09 13:19:34 at com.linkedin.metadata.kafka.MetadataChangeProposalsProcessor.consume(MetadataChangeProposalsProcessor.java:63)
2024-01-09 13:19:34 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-01-09 13:19:34 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2024-01-09 13:19:34 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-01-09 13:19:34 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2024-01-09 13:19:34 at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169)
2024-01-09 13:19:34 at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
2024-01-09 13:19:34 at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:56)
2024-01-09 13:19:34 at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:347)
2024-01-09 13:19:34 at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92)
2024-01-09 13:19:34 at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:53)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2670)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2650)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2577)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2457)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2335)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2006)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1375)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1366)
2024-01-09 13:19:34 at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1257)
2024-01-09 13:19:34 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2024-01-09 13:19:34 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2024-01-09 13:19:34 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2024-01-09 13:19:34 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2024-01-09 13:19:34 at java.base/java.lang.Thread.run(Thread.java:829)
2024-01-09 13:19:34 Caused by: com.linkedin.data.template.RequiredFieldNotPresentException: Field "value" is required but it is not present
2024-01-09 13:19:34 at com.linkedin.data.template.RecordTemplate.obtainValueOrDefault(RecordTemplate.java:511)
2024-01-09 13:19:34 at com.linkedin.data.template.RecordTemplate.obtainDirect(RecordTemplate.java:391)
2024-01-09 13:19:34 at com.linkedin.data.template.DynamicRecordTemplate.getValue(DynamicRecordTemplate.java:111)
2024-01-09 13:19:34 at com.linkedin.restli.common.ActionResponse.getValue(ActionResponse.java:83)
2024-01-09 13:19:34 at com.linkedin.restli.internal.client.ActionResponseDecoder.wrapResponse(ActionResponseDecoder.java:64)
2024-01-09 13:19:34 at com.linkedin.restli.internal.client.RestResponseDecoder.createResponse(RestResponseDecoder.java:177)
2024-01-09 13:19:34 at com.linkedin.restli.internal.client.RestResponseDecoder.decodeResponse(RestResponseDecoder.java:166)
2024-01-09 13:19:34 at com.linkedin.restli.internal.client.ExceptionUtil.exceptionForThrowable(ExceptionUtil.java:94)
2024-01-09 13:19:34 at com.linkedin.restli.client.RestLiCallbackAdapter.convertError(RestLiCallbackAdapter.java:66)
2024-01-09 13:19:34 at com.linkedin.common.callback.CallbackAdapter.onError(CallbackAdapter.java:86)
2024-01-09 13:19:34 at com.linkedin.r2.message.timing.TimingCallback.onError(TimingCallback.java:81)
2024-01-09 13:19:34 at com.linkedin.r2.transport.common.bridge.client.TransportCallbackAdapter.onResponse(TransportCallbackAdapter.java:47)
2024-01-09 13:19:34 at com.linkedin.r2.filter.transport.FilterChainClient.lambda$createWrappedClientTimingCallback$0(FilterChainClient.java:125)
2024-01-09 13:19:34 at com.linkedin.r2.filter.transport.ResponseFilter.onRestError(ResponseFilter.java:79)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedRestFilter.onRestError(TimedRestFilter.java:96)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:166)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:132)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator.onError(FilterChainIterator.java:101)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedNextFilter.onError(TimedNextFilter.java:48)
2024-01-09 13:19:34 at com.linkedin.r2.filter.compression.ClientCompressionFilter.onRestError(ClientCompressionFilter.java:286)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedRestFilter.onRestError(TimedRestFilter.java:96)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:166)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:132)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator.onError(FilterChainIterator.java:101)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedNextFilter.onError(TimedNextFilter.java:48)
2024-01-09 13:19:34 at com.linkedin.r2.filter.message.rest.RestFilter.onRestError(RestFilter.java:84)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedRestFilter.onRestError(TimedRestFilter.java:96)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:166)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:132)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator.onError(FilterChainIterator.java:101)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedNextFilter.onError(TimedNextFilter.java:48)
2024-01-09 13:19:34 at com.linkedin.r2.filter.message.rest.RestFilter.onRestError(RestFilter.java:84)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedRestFilter.onRestError(TimedRestFilter.java:96)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:166)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:132)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator.onError(FilterChainIterator.java:101)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedNextFilter.onError(TimedNextFilter.java:48)
2024-01-09 13:19:34 at com.linkedin.r2.filter.message.rest.RestFilter.onRestError(RestFilter.java:84)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedRestFilter.onRestError(TimedRestFilter.java:96)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:166)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnError(FilterChainIterator.java:132)
2024-01-09 13:19:34 at com.linkedin.r2.filter.FilterChainIterator.onError(FilterChainIterator.java:101)
2024-01-09 13:19:34 at com.linkedin.r2.filter.TimedNextFilter.onError(TimedNextFilter.java:48)
2024-01-09 13:19:34 at com.linkedin.r2.filter.transport.ClientRequestFilter.lambda$createCallback$0(ClientRequestFilter.java:102)
2024-01-09 13:19:34 at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:82)
2024-01-09 13:19:34 at com.linkedin.r2.transport.http.client.rest.ExecutionCallback.lambda$onResponse$0(ExecutionCallback.java:64)
2024-01-09 13:19:34 ... 3 more```