Troubleshooting Kafka Ingestion in DataHub UI 0.12.4

ingestion
1

Original Slack Thread

Hello All,

I am trying to setup kafka ingestion through datahub UI (0.12.4 - create new datasource service)
however, while running this configuration is failing with the following error.

    topics = self.consumer.list_topics(
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}```
I am able to connect to the kafka broker host and port using other tools and it is working. I am able to see the topics.
can someone tell me what could be the issue here. I am new to datahub![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06KT9ELZQX/image.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

is kafka running on the same host as the datahub instance?

4

Hello Anton,
Thanks for the response.

Yes. both are on the same host.

5

another thing to check if make sure that 29092 the port accepting plain-text connection

6

yes. verified that the above port is accepting plain-text connection.

7 

 run_id: 'urn:li:dataHubExecutionRequest:8c1431ff-c112-4901-99b7-5fb858755bf0'
source:
  type: kafka
  config:
    connection:
      consumer_config:
        security.protocol: PLAINTEXT
      bootstrap: 'localhost:29092'
    stateful_ingestion:
      enabled: false
pipeline_name: 'urn:li:dataHubIngestionSource:ad9a9fea-776b-4d13-835a-cc504ba7e5f2'```
8

it looks like kafka/network issue to me. Try to connect from the ingestion container to the kafka port with e.g. telnet/nc and confirm that kafka is indeed reachable from there. For example, it may be reachable from the host, but not from the container.

9

Thanks <@U063WTWMH2L> for the reply. it worked. it was the problem with the network mapping.
i have one more query.
I am able to bind the kafka topic in datahub ui (ingestion), but i am seeing below error.

'The schema registry subject for the value schema is not found. The topic is either schema-less, or no messages have been written to the topic yet.

is it required to configure schema registry ? cant i send schema less data to the topic? if that is the case, how can i see data in the datahub to tag or to map under domain?

Thanks in advance for the support.

attachment
attachment1275×354 11.2 KB