Troubleshooting Kafka Setup on Datahub Moving to EKS Fargate

Original Slack Thread

Hi Everyone, I’m running Datahub v0.11 and have a working Datahub install with EKS and NodeGroups (Ec2).

I’m trying to move the entire cluster to EKS Fargate (serverless). Kafka and Zookeeper need persistent storage, and I’ve set up and EFS file system for those two pods.

The pre-requisites pods start and stay running, but Kafka crashes from time to time. When I run the datahub helms, I keep seeing this error in the logs and the kafka setup job keeps failing. Does anyone have any troubleshooting ideas?

[2024-02-21 22:46:41,386] WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker prerequisites-kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
Exception in thread "main" java.lang.RuntimeException: Request METADATA failed on brokers List(prerequisites-kafka:9092 (id: -1 rack: null))at kafka.admin.BrokerApiVersionsCommand$AdminClient.sendAnyNode(BrokerApiVersionsCommand.scala:156)
at kafka.admin.BrokerApiVersionsCommand$AdminClient.findAllBrokers(BrokerApiVersionsCommand.scala:179)
at kafka.admin.BrokerApiVersionsCommand$AdminClient.awaitBrokers(BrokerApiVersionsCommand.scala:171)
at kafka.admin.BrokerApiVersionsCommand$.execute(BrokerApiVersionsCommand.scala:61)
at kafka.admin.BrokerApiVersionsCommand$.main(BrokerApiVersionsCommand.scala:55)
at kafka.admin.BrokerApiVersionsCommand.main(BrokerApiVersionsCommand.scala)

it looks like kafka brokers are not running, or not accepting connections. check logs in the kafka pod

Thank you, it’s working now