Troubleshooting SSL Error in Nifi Setup for Datahub Ingestion

ingestion
1

Original Slack Thread

Hello,

I’m trying to setup an ingestion on Datahub with Nifi, And I’m having the following error:

<https://nifi-console-dev01-nifi.apps/nifi/> <class 'requests.exceptions.SSLError'>: HTTPSConnectionPool(host='nifi-console-dev01-nifi.apps', port=443): Max retries exceeded with url: /nifi-api/flow/about (Caused by SSLError(SSLError(0, 'Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:811)')))

My setup is the following:

    type: nifi
    config:
        site_url: '<https://nifi-console-dev01-nifi.apps/nifi/>'
        auth: CLIENT_CERT
        ca_file: /nifi/ca.crt
        client_cert_file: /nifi/tls.crt
        client_key_file: /nifi/tls.key```
The ca certificate is valid, if I'm trying to check with a curl providing the ca it works properly.
2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

The error you’re encountering is related to SSL verification when trying to connect to your NiFi instance. This issue often arises when there is a mismatch in SSL configurations, such as using a self-signed certificate or incorrect SSL context settings.

Based on the <Slack discussion>, here are some steps you can take to resolve the issue:

  1. SSL Verification: If you are using a self-signed certificate, you might need to disable SSL verification for testing purposes. You can do this by setting the ca_file configuration to False in your DataHub configuration file. This will bypass SSL certificate validation.

  2. Python Version Compatibility: There is a known issue with Python 3.10 when creating the SSL context with CLIENT_AUTH. If you are using Python 3.10, consider modifying the SSL context to use SERVER_AUTH instead. This change has been reported to work successfully with Python 3.9.

  3. Configuration Example: Ensure your configuration file for NiFi ingestion is correctly set up. Here is an example configuration snippet:

    source:
  type: nifi
  config:
    connect_uri: 'https://nifi-console-dev01-nifi.apps'
    ssl: true
    ca_file: False  # Disable SSL verification for testing

  4. Testing with CLI: If you are using both CLI and UI for ingestion, test the configuration with the CLI first to ensure it works before applying it to the UI.

Would you like more detailed guidance on any specific part of the configuration or troubleshooting steps?

Sources:

  • <Slack discussion on SSL Configuration> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
4

<@U06TM7M40RK> where should I change SERVER_AUTH ?

5

To change the SSL context from CLIENT_AUTH to SERVER_AUTH, you will need to modify the code in the nifi.py file of the DataHub ingestion source for NiFi. This change is necessary if you are encountering issues with Python 3.10, as mentioned in the <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/nifi.py|GitHub file>.

Here’s how you can make the change:

  1. Locate the nifi.py File: This file is part of the DataHub ingestion source code. You can find it in the metadata-ingestion/src/datahub/ingestion/source/ directory of the DataHub project.

  2. Modify the SSL Context: In the nifi.py file, locate the section where the SSL context is created. You will find a line similar to this:

    self.context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)

  3. Change to SERVER_AUTH: Modify the line to use SERVER_AUTH instead of CLIENT_AUTH:

    self.context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)

  4. Rebuild and Test: After making this change, rebuild your DataHub ingestion setup and test the connection to your NiFi instance to ensure that the issue is resolved.

Would you like more detailed instructions on how to rebuild the DataHub ingestion setup after making this change?

Sources:

6

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><