Resolving issues with secrets in Kubernetes and querying secrets with GraphQL in DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> the secret can not be resolved after pods restarted, why ?

2

The issue of secrets not being resolved after pods are restarted is a common problem in Kubernetes deployments, particularly with DataHub. This can occur due to several reasons:

  1. Secret Persistence: According to <Slack Slack Discussion on Secret Persistence>, secrets might not persist correctly across pod restarts or redeployments. Secrets are typically stored in the container and not in a database, which could lead to loss upon redeployment. To address this, ensure that secrets are properly managed and persisted across deployments. You might need to use a more robust secret management system integrated with Kubernetes, such as Kubernetes Secrets or an external secret manager like HashiCorp Vault.

  2. Encryption Key Issue: As mentioned in <Slack Slack Thread on Encryption Key Issue>, GMS (Graph Metadata Service) auto-generates an encryption key used for encrypting secrets. If GMS is redeployed, the encryption key might change, causing previously input secrets to become inaccessible. To prevent this, provide the encryption key yourself by updating the Helm chart. Ensure that the encryption key is consistent across deployments.

  3. Environment Variables: If environment variables are being used for secrets and tokens, ensure they are correctly passed to all relevant pods. Verify that the environment variables are correctly set in your Helm charts and Kubernetes manifests. Check that these variables are available in all pods involved in the ingestion process.

  4. Secret Creation and Reference: Ensure that the secret is created and referenced correctly in the ingestion recipe. Follow the steps for creating and referencing secrets as per the <https://datahubproject.io/docs/0.13.1/ui-ingestion#creating-a-secret|DataHub documentation>.

Would you like more detailed steps or further assistance on any specific part of the process?

Sources:

  • <Slack Slack Discussion on Secret Persistence> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
3

<@U06TM7M40RK> can you provide me query to be able to query the secret from graphQL ?

4

To query secrets using GraphQL in DataHub, you can use the getSecretValues query. This query requires the MANAGE_SECRETS privilege to retrieve the plaintext values of secrets. Here’s an example of how you can structure your GraphQL query:

query getSecretValues($input: GetSecretValuesInput!) {
  getSecretValues(input: $input) {
    name
    value
  }
}

To execute this query, you need to provide the GetSecretValuesInput which includes a list of secret names you want to retrieve. Here’s an example of how you might call this query with variables:

{
  "input": {
    "secrets": ["SECRET_NAME_1", "SECRET_NAME_2"]
  }
}

This will return the names and plaintext values of the specified secrets.

Please ensure that you have the necessary permissions to execute this query, as it requires the MANAGE_SECRETS privilege.

Would you like more detailed steps or further assistance on any specific part of the process?

Sources:

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><