<@U06TM7M40RK> use secret in ingestion recipe
To use secrets in an ingestion recipe with DataHub, you can follow these steps:
-
Create a Secret in DataHub: First, navigate to the ‘Secrets’ tab in the DataHub UI and click on ‘Create new secret’. Provide a unique name for the secret, the value to be encrypted, and an optional description. This will securely store the secret within DataHub’s storage layer. For example, you might create a secret for a database password.
-
Reference the Secret in Your Recipe: Once the secret is created, you can reference it in your ingestion recipe using variable substitution. For example, if you created a secret named
MYSQL_PASSWORD
, you can reference it in your recipe as${MYSQL_PASSWORD}
.Here’s an example of how you might define a recipe using secrets:
source: type: mysql config: host_port: "localhost:3306" database: my_db username: ${MYSQL_USERNAME} password: ${MYSQL_PASSWORD} include_tables: true include_views: true profiling: enabled: true sink: type: datahub-rest config: server: "http://datahub-gms:8080"
-
Execution: When the ingestion source with this recipe executes, DataHub will resolve the secrets by substituting the placeholders with their decrypted values. Secret values are not persisted to disk beyond execution time and are never transmitted outside DataHub.
For more detailed information, you can refer to the <https://github.com/datahub-project/datahub/blob/master/docs/ui-ingestion.md|DataHub UI Ingestion Documentation> and the <https://datahubproject.io/docs/0.13.1/ui-ingestion#creating-a-secret|DataHub documentation on creating and referencing secrets>.
Would you like more detailed steps or have any specific questions about this process?
Sources:
- <https://github.com/datahub-project/datahub/blob/master/docs/ui-ingestion.md|DataHub UI Ingestion Documentation>
- <https://datahubproject.io/docs/0.13.1/ui-ingestion#creating-a-secret|DataHub documentation on creating and referencing secrets>
0 button
0 button
Hint: Mention @runllm in the thread for followups.
Hi <@U08PVDCPJAG>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.