Managing Secrets for DataHub CLI Ingestion via Environment Variables and External Tools

ingestion
1

Original Slack Thread

<@U06TM7M40RK> Hi! Im attempting to add an ingestion via CLI, but I am struggling to refer to the secret that I defined through UI. How come?

2

When using the DataHub CLI for ingestion, referencing secrets that were defined through the UI can be a bit tricky because the CLI does not directly access the secrets stored in the DataHub UI. Instead, you need to use environment variables or an external secret management system to handle secrets in the CLI.

Here are some steps you can follow to manage secrets when using the CLI:

  1. Use Environment Variables: You can set environment variables in your local or session environment to store secret values. These environment variables can then be referenced in your ingestion recipe.

  2. External Secret Management: Consider using an external secret management tool like AWS Secrets Manager or HashiCorp Vault. Retrieve these secrets in your CLI session and set them as environment variables before running your ingestion command.

  3. Ingestion Recipe: Ensure your ingestion recipe is set up to use these environment variables. For example, if you have a secret for a MySQL username, your recipe might look like this:

    source:
    type: mysql
    config:
        host_port: 'localhost:3306'
        database: my_db
        username: ${MYSQL_USERNAME}
        password: ${MYSQL_PASSWORD}
        include_tables: true
        include_views: true
        profiling:
            enabled: true
sink:
    type: datahub-rest
    config:
        server: 'http://datahub-gms:8080'

  4. Set Environment Variables: Before running your CLI command, set the necessary environment variables in your terminal session:

    export MYSQL_USERNAME='your-username'
export MYSQL_PASSWORD='your-password'

According to the <Slack Slack discussion>, using environment variables or an external secret store is recommended for handling secrets in CLI-based ingestion.

Would you like more detailed steps or have any specific questions about this process?

Sources:

3

Hi <@U07RYKJ49LY>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

4

OK for now, I’m just setting the secret variables locally in shell session. However a way to access/utilize the secrets that are set in the UI would be most welcome.

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><