Configuring SSL Certificate for Ingestion from Superset in DataHub

ingestion
1

Original Slack Thread

Hi. how to configure ingestion from a superset with a certificate issued by a domain controller. Help pls. how and what to do step by step to add a certificate to the trusted ones

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: We’re experimenting with a new AI assistant for questions in the <#CUMUWQU66> channel to help you track down helpful resources more quickly. Keep in mind that responses may not always be spot-on; feel free to ask it follow-up questions and don’t forget to provide its responses a :thumbsup: or :thumbsdown: so other folks in the Community know if you’re still looking for help on the topic!

3

To help you more effectively, could you please provide the following information?

  1. Are you using the UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g., 0.12.0)?
  3. What data source(s) are you integrating with DataHub? (e.g., Superset)?

Once I have this information, I can provide a more tailored step-by-step guide to help you configure ingestion from Superset with a certificate issued by a domain controller.

Sources:

4

UI ingestion, acryl-datahub, version 0.13.2.4

5

export REQUESTS_CA_BUNDLE=""
not working

6

disable_ssl_verification: bool = True
not working…

7

<@U06TM7M40RK> are there any other options on how to do this?

8

Thank you for the additional information. Since you are using the UI for ingestion with DataHub version 0.13.2.4 and integrating with Superset, here are the steps you can take to address the SSL certificate verification issue with a certificate issued by a domain controller:

Step-by-Step Guide

  1. Obtain the CA Certificate:

    • Ensure you have the CA certificate issued by your domain controller. This certificate should be in .pem format.

  2. Add CA Certificate to Trusted Certificates:

    • You need to add this CA certificate to the trusted certificates on the machine where DataHub is running.

  3. Set the REQUESTS_CA_BUNDLE Environment Variable:

    • Set the REQUESTS_CA_BUNDLE environment variable to point to the CA certificate file.
    export REQUESTS_CA_BUNDLE=/path/to/your/ca_certificate.pem

  4. Configure Superset Source in DataHub UI:

    • In the DataHub UI, configure the Superset source. You need to specify the ca_certificate_path parameter in your ingestion configuration.

    Example configuration:

    source:
  type: superset
  config:
    connect_uri: 'https://your-superset-server/login/'
    password: '${SUPERSET_PASSWORD}'
    provider: db
    username: 'your-username'
    ca_certificate_path: '/path/to/your/ca_certificate.pem'

  5. Restart DataHub Services:

    • After setting the environment variable and configuring the source, restart your DataHub services to ensure the changes take effect.

Example Configuration in DataHub UI

If you are configuring this through the UI, you will need to locate the corresponding fields for ca_certificate_path and set them accordingly. The UI should have options to provide a path to the CA certificate.

Additional Resources

For more details, you can refer to the <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/superset.py|Superset Source Configuration> and <Superset | DataHub Ingestion Configuration>.

Would you like more detailed instructions on any of these steps or have any specific questions about these steps?

Sources:

9

<@U06TM7M40RK> how to Restart DataHub Services?

10

To restart DataHub services, you typically need to restart the Docker containers running the DataHub components. Here’s how you can do it step-by-step:

Step-by-Step Guide to Restart DataHub Services

  1. Navigate to Your DataHub Directory:

    • Go to the directory where your docker-compose.yml file is located. This file is usually in the directory where you set up DataHub.
    cd /path/to/your/datahub/directory

  2. Stop DataHub Services:

    • Use the following command to stop all DataHub services:
    docker-compose down

  3. Start DataHub Services:

    • After stopping the services, start them again using the following command:
    docker-compose up -d

    The -d flag runs the containers in detached mode, allowing you to continue using your terminal.

  4. Verify Services Are Running:

    • You can check the status of the containers to ensure they are running correctly:
    docker-compose ps

Example Commands

Here’s a consolidated example of the commands you would run:

cd /path/to/your/datahub/directory
docker-compose down
docker-compose up -d
docker-compose ps

Additional Notes

  • Custom Docker Compose File: If you are using a custom Docker Compose file, make sure to specify it with the -f option:

    docker-compose -f custom-docker-compose.yml down
docker-compose -f custom-docker-compose.yml up -d

  • Logs: To view logs and troubleshoot any issues, you can use:

    docker-compose logs -f

Would you like more detailed instructions on any of these steps or have any specific questions about restarting DataHub services?

Sources:

11

<@U06TM7M40RK> how to restert datahub with quickstart

12

To restart DataHub using the quickstart method, you can use the datahub docker quickstart command. Here’s how you can do it step-by-step:

Step-by-Step Guide to Restart DataHub with Quickstart

  1. Stop DataHub Services:

    • Use the following command to stop the DataHub services:
    datahub docker quickstart --stop

  2. Start DataHub Services:

    • After stopping the services, start them again using the quickstart command:
    datahub docker quickstart

Example Commands

Here’s a consolidated example of the commands you would run:

datahub docker quickstart --stop
datahub docker quickstart

Additional Notes

  • Environment Variables: Ensure that any necessary environment variables (e.g., REQUESTS_CA_BUNDLE) are set before starting the services.

  • Custom Compose File: If you are using a custom Docker Compose file, you can specify it with the --quickstart-compose-file option:

    datahub docker quickstart --quickstart-compose-file /path/to/custom-compose.yml

Example with Environment Variables

If you need to set environment variables before restarting, you can do it like this:

export REQUESTS_CA_BUNDLE=/path/to/your/ca_certificate.pem
datahub docker quickstart --stop
datahub docker quickstart

Would you like more detailed instructions on any of these steps or have any specific questions about restarting DataHub using the quickstart method?

Sources: