Issues with Auth/Encryption Secrets in Helm Charts Deployment

all-things-deployment
1

Original Slack Thread

I believe I’m running into issues with auth/encryption secrets deployed using the helm charts. Somehow things are getting broken upon updating helm charts.
Some of the symptoms

  1. Access tokens which I generated and set to never expire are being rejected by GMS after a deployment. Newly generated tokens work fine
  2. Sometimes GMS starts rejecting frontend login requests. I have to kill both the GMS and frontend pods for this to resolve
    Looking at the deployed secrets it doesn’t appear like they’re being changed, but is it recommended to just set them manually to avoid them accidentally changing? Anyone have similar issues?
2

Another similar error when trying to generate signup links as an admin. Login workedattachment

3

Just ran a deployment on our staging cluster and saw that the secrets changed before/after for datahub-auth-secrets

4

Hi Drew, can you share more on what updates you’re making to the helm charts?

5

Typically just configmap updates with new ingestion recipes. I think I mostly solved this issue by moving most of the secrets into our external secrets manager.