Troubleshooting SSL Configuration and Kafka Topic Error in SystemUpdate Job

all-things-deployment
1

Original Slack Thread

Hi Team - I am seeing the following errors in our systemUpdate job

org.apache.kafka.common.errors.TimeoutException: Topic DataHubUpgradeHistory_v1 not present in metadata after 60000 ms.```
Here is our ssl configuration. Please let me know if this needs to be added to the datahubSystemUpdate section or not?
```          credentialsAndCertsSecrets:
            name: datahub-certs
            path: /ssl
            secureEnv:
              ssl.truststore.password: password
              kafkastore.ssl.truststore.password: password
    
          springKafkaConfigurationOverrides:
            ssl.truststore.location: /ssl/truststore.jks
            kafkastore.ssl.truststore.location: /ssl/truststore.jks
            sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="datahub_staging" password="${jaasconfig}";
            security.protocol: SASL_SSL
            kafkastore.security.protocol: SASL_SSL
            ssl.protocol: TLS
            ssl.truststore.type: JKS
            sasl.mechanism: SCRAM-SHA-256
            ssl.endpoint.identification.algorithm: ```
2

when I look into the systemUpdate job log, I see the following are completely empty

        ssl.truststore.location = null
        ssl.truststore.password = null```
Can someone help in providing pointers on how to pass this onto the systemUpdate job?
3

<@U02TYQ4SPPD> <@U01JUBAPZPS> any suggestions?

4

I confirmed by logging into our pod that the truststore file is present . but not sure why the configs are all in the log.

5

FYI
sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="datahub_staging" password="${jaasconfig}";
In this line jaasconfig is injected an env variable. into the pod

6

Hi <@U05KVNAL068> are you sure that DataHubUpgradeHistory_v1 topic does not exist on your Kafka cluster?
How do you install your DataHub today? Helm charts or docker setup?
Have you checked kafka-setup-job logs? DataHubUpgradeHistory_v1 should have been created by this setup job.

7

hi <@U040V3TUD5J> - we have a managed kafka cluster and because of that, we explicitly created the kafka topics and disabled the kafka setup job.
the rest of the datahub installation is all done via helm charts.

8

i think the issue might be something to do with the ssl configuration.

9

org.apache.kafka.common.errors.TimeoutException: Topic DataHubUpgradeHistory_v1 not present in metadata after 60000 ms.
This could mean that the KafkaHealthChecker is not able to contact the broker if the topic exists.
How does the describe command output looks on the systemUpdateJob?
kubectl describe job datahub-system-update-job

10

I assume you have seen this page: https://datahubproject.io/docs/0.10.5/how/kafka-config

11

Super helpful Upendra. Thanks for your help

12

I will go through the docs