Using Secrets in an Ingestion Recipe with DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> use secret in ingestion recipe

2

To use secrets in an ingestion recipe with DataHub, you can follow these steps:

  1. Create a Secret in DataHub: First, navigate to the ‘Secrets’ tab in the DataHub UI and click on ‘Create new secret’. Provide a unique name for the secret, the value to be encrypted, and an optional description. This will securely store the secret within DataHub’s storage layer. For example, you might create a secret for a database password.

  2. Reference the Secret in Your Recipe: Once the secret is created, you can reference it in your ingestion recipe using variable substitution. For example, if you created a secret named MYSQL_PASSWORD, you can reference it in your recipe as ${MYSQL_PASSWORD}.

    Here’s an example of how you might define a recipe using secrets:

    source:
  type: mysql
  config:
    host_port: "localhost:3306"
    database: my_db
    username: ${MYSQL_USERNAME}
    password: ${MYSQL_PASSWORD}
    include_tables: true
    include_views: true
    profiling:
      enabled: true
sink:
  type: datahub-rest
  config:
    server: "http://datahub-gms:8080"

  3. Execution: When the ingestion source with this recipe executes, DataHub will resolve the secrets by substituting the placeholders with their decrypted values. Secret values are not persisted to disk beyond execution time and are never transmitted outside DataHub.

For more detailed information, you can refer to the <https://github.com/datahub-project/datahub/blob/master/docs/ui-ingestion.md|DataHub UI Ingestion Documentation> and the <https://datahubproject.io/docs/0.13.1/ui-ingestion#creating-a-secret|DataHub documentation on creating and referencing secrets>.

Would you like more detailed steps or have any specific questions about this process?

Sources:

3

Hi <@U08PVDCPJAG>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.