Troubleshooting Kafka Connection Issue: SSL Listener Configuration Error

ingestion
1

Original Slack Thread

http://b-2.vhmvinclubstagmsk.lggqx5.c5.kafka.ap-southeast-1.amazonaws.com:9096/bootstrap|b-2.vhmvinclubstagmsk.lggqx5.c5.kafka.ap-southeast-1.amazonaws.com:9096/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY, 4 identical error(s) suppressed)

i got this error in datahub-action, but in gui still good

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

The error message you’re encountering, “Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10,” typically indicates a misconfiguration in the security protocol settings for your Kafka connection. This is a common issue when connecting to Kafka brokers that require SSL/TLS encryption.

Here are some steps and configurations that might help resolve the issue:

  1. Verify Security Protocol Configuration: Ensure that the security protocol configuration is correctly set. If you are using SSL, your configuration should specify security.protocol as SSL.

  2. Check Kafka Broker Version: Ensure that your Kafka broker version is indeed greater than 0.10, as the error message suggests.

  3. Custom Executor Configuration: If you are using DataHub Actions, you might need to customize the executor.yaml file to include the correct Kafka configurations. Here is an example configuration:

    name: "ingestion_executor"
source:
  type: "kafka"
  config:
    connection:
      bootstrap: "<your_kafka_bootstrap_server>"
      consumer_config:
        security.protocol: "SSL"
        ssl.ca.location: "/path/to/ca.pem"
        ssl.certificate.location: "/path/to/client.pem"
        ssl.key.location: "/path/to/key.pem"

  4. Environment Variables: Ensure that all necessary environment variables are correctly set, especially those related to Kafka security settings.

  5. Check for Additional Configuration: If you are using AWS MSK, ensure that you have configured the necessary IAM roles and policies if using IAM authentication.

If you need more detailed guidance on any of these points, please let me know!

Sources:

4

my protocol is SASL_SSL

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><