DatahubGMS Issues: Mysql Database Disk Full, Queries Not Working

Original Slack Thread

HELP, DatahubGMS not accepting queries anymore after an issue on the Mysql database (disk full). I don’t see anything in the logs except GMS saying that Elastic search is currently rebuilding its indices, but I don’t think it is a blocking task. I am really clueless on what is going on as it use to run perfectly for 2 years now and tomorrow afternoon I am supposed to make a demo of the product! :cry:. Could it be related to some Mysql database corruption? Thanks in advance for your assistance. Laurent

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!

What errors are you seeing? It’s unclear what not accepting queries means.

Can you also be a bit more specific about what ElasticSearch is rebuilding its indices means? It’s certainly possible that this would block queries from executing successfully. Having the precise errors you’re getting would help narrow this down.

It’s possible that there are additional issues with MySQL depending on what all happened while it ran out of disk space and how it was recovered.

Thanks Ryan for your message. Actually GMS was able to restart as soon as the re - index process of ES finished; it might have detected some mismatch between ES and MySQL and required a re-alignment?