Troubleshooting Cloud SQL password authentication issue with Kubernetes secret in Datahub deployment

all-things-deployment
1

Original Slack Thread

Hi there :wave:

I am trying to deploy Datahub to GKE.
I chose to use Cloud SQL to host my postgres Database.

However, the postgresql-setup-job fails if I gives it the password in a secret instead of passing it as a value

In the yml, if I set

    datasource:
      ...
      username: datahub
      password:
        value: myPassword```
everything works

If on the contrary I set it like that
```  sql:
    datasource:
      ...
      username: datahub
      password:
        secretRef: postgres-secret
        secretKey: fmdp-psql-password```
I always get the following error:
`psql: error: connection to server at "prerequisites-gcloud-sqlproxy" (10.28.9.85), port 3306 failed: FATAL:  password authentication failed for user "datahub"`

Anyone has a clue about why it does that and how to fix this issue?

Thanks! :pray:
2

Several checks I have done :
• My password is base64 encoded in the kube secret
• The environment variable is correctly set attachment

3

<@U03MF8MU5P0> any idea on this?

4

Hello <@U0670CBQ5U6>, how do you create the secret? Could it be related to the encoding? If I’m not mistaken, the secrets are stored unencrypted in k8s. Have you considered using an external provider? For example we’re using AKS which integrates with Azure Key Vault and keep the secrets there.

5

Hi!

I create the secret using kube, just the same as for other services. I get the problem only with postgresql. I did not try with a vault for now

6

So if I look at your yaml file, the command to create the secret would be: kubectl create secret generic postgres-secret --from-literal=fmdp-psql-password=&lt;your password in plain text here&gt; Could you confirm? Maybe first delete and re-create it just in case?

7

To be exact, I create it using a yml definition and running kubectl apply -f postgres_secret.yml

kind: Secret
metadata:
  name: postgres-secret
type: Opaque
data:
  fmdp-psql-password: &lt;base64 encoded password&gt;```
I have tried several time to change the password, delete and re-create the secret.

It still does not work :disappointed:
8

What response do you get when running the command I provided?

9

Looks like it works!! :tada:

10

I don’t get why, because there should be no difference from the way I created it but weirdly, it works

11

Happy that it works. It works because you created the secret in plain text. Otherwise the password is sent in it’s base64 encoded representation and the database rejects it.

12

Ok, thank you very much!