Using Secrets from `/mnt/secrets` in DataHub Ingestion Recipe with Datahub-Actions

ingestion
1

Original Slack Thread

<@U06TM7M40RK> how to use secret from ingestion recipe for /mnt/secrets mounted in datahub-actions

2

To use secrets from a mounted directory like /mnt/secrets in your DataHub ingestion recipe, you can follow these steps:

  1. Mount the Secret: Ensure that your Kubernetes setup mounts the secret to the /mnt/secrets directory in the datahub-actions pod. This is typically done using a Kubernetes Secret or a CSI driver for secret management like AWS Secrets Manager or Azure Key Vault.

  2. Access the Secret in the Recipe: In your ingestion recipe, you can reference the secret by reading the file directly from the mounted path. This involves using the __DATAHUB_TO_FILE_<property> directive in your recipe to specify that the value should be read from a file.

Here’s an example of how you might configure your ingestion recipe to use a secret stored in a file:

source:
  type: mysql
  config:
    host_port: "localhost:3306"
    database: "my_db"
    username: "root"
    # Use the __DATAHUB_TO_FILE directive to read the password from a file
    __DATAHUB_TO_FILE_password: "/mnt/secrets/mysql-password"
    include_tables: true
    include_views: true
    profiling:
      enabled: true
sink:
  type: datahub-rest
  config:
    server: "http://datahub-gms:8080"

In this example, the password is read from the file located at /mnt/secrets/mysql-password. The __DATAHUB_TO_FILE_password directive tells DataHub to read the password from the specified file path.

  1. Ensure Proper Permissions: Make sure that the datahub-actions pod has the necessary permissions to read from the /mnt/secrets directory. This might involve setting the correct file permissions and ensuring that the Kubernetes service account has access to the secret.

Would you like more detailed steps or further assistance on any specific part of the process?

Sources:

3

Hi <@U08PVDCPJAG>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.