Troubleshooting docker-compose deployment issues with Kafka configuration

all-things-deployment
1

Original Slack Thread

Hi all, I’m having some trouble with the deployment using docker-compose and an external kafka system which is also deployed using docker. I updated the new host for kafka in the compose file, but receiving the error that datahub-gms is unhealthy. This is what I got from the log of gms and datahub-upgrade:

at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:930)
at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1386)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:962)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:74)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1135)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1023)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:548)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:324)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
at java.base/java.lang.Thread.run(Thread.java:829)```
Somehow changing the host variables in the compose file didnt reflect here, even initially the gms had popped out that it could connect to kafka stack `Connected to <<host>>:29092` . I think the host might be hard-coded in the source code. Could anyone please help with this?
2

Hi Which compose file did you use for datahub deployment and where did you update the broker address?

3

Hi I’m using this file with broker, kafka-setup, schema-registry, zookeeper removed. https://github.com/acryldata/datahub/blob/master/docker/quickstart/docker-compose.quickstart.yml
I updated KAFKA_BOOTSTRAP_SERVER, SCHEMA_REGISTRY_URL, KAFKA_SCHEMAREGISTRY_URL for datahub-actions, datahub-frontend-react, datahub-gms and datahub-upgrade

4

Have you changed KAFKA_BOOTSTRAP_SERVER=broker:29092

as per your config in all env sections?

5

Yes, I have. The components that I mentioned above are all having the variable

6

seems it related to KAFKA_ADVERTISED_LISTENERS var from kafka stack
https://www.confluent.io/blog/kafka-listeners-explained/

7

Is it possible to share the compose file? I am not sure what is the wrong config.

8

the config is in kafka compose file not in datahub. You can see on the page, to connect to the broker from an external client (here is datahub), it will use KAFKA_ADVERTISED_LISTENERS. Kafka will send the client the host defined in KAFKA_ADVERTISED_LISTENERS, and client will try to connect using this host. So in my case, I only updated the host in datahub compose file, so that why initially gms can connect to kafka but later it threw the UnknowHostException since in kafka compose file KAFKA_ADVERTISED_LISTENERS was still “broker:29092”