Resolving MySQL startup probe issue by increasing PVC size and redeployment

Original Slack Thread

Hello <@U0227B8D97F>, I had a deployed working version of DataHub v0.10.5.5. After tinkering with several config setting for SSO, GMS authentication. I decided to get it back to the a normal state as deployed by my helm charts. I am getting issues in re-deployment on the prerequisites. More specifically mysql - startup probe. I get the below error. I am guessing it has something to do with the pvc, but not sure. I dont want to nuke everything as I have things I do not want to lose. Any guidance will be appreciated.

I have figured out the issue, so I thought I would post it here so if others had the same issue, they could use this as a knowledge base as well.

The mysql start up was trying to find a socket file as defined in the cnf file. So I opened up a terminal within the pod (deployed on AKS), and ran the ps -aux command. I found the mysql command that it was trying to run, ran it manually with -x flag to get verbose logging. I saw the error was related to disk size. I ran the command df -h to double check and sure enough it was out of disk storage.

To resolve: I increased the PVC size and redeployed and it is now working.

Hope someone else will find this useful.

Thank you so much for sharing this in the community! :slight_smile:

Hey <@U05HMP51HLJ>, any tips on how you can resize the pvc? I can’t find any docs on how to do that.

<@U04P50Z1YBY>, sorry for the late reply here. I have updated my values.yaml for custom values and redeployed the pre-requisites for my-sql. You can increase manually inside kubernetes, the instructions will vary depending on which cluster environment you are using. But if you don’t have this persistence config in your values.yaml, it will revert back and you will continue to have issues.

  enabled: true
  primary:
    resources:
      requests:
        cpu: 100m
        memory: 1024Mi
      limits:
        cpu: 200m
        memory: 2048Mi
    persistence:
      size: 16Gi```

Thanks Christopher, appreciate the help!