Troubleshooting GMS Startup Issue with DataHub Upgrade

troubleshoot
1

Original Slack Thread

Hi Folks,
After restarting the quickstart GMS is not starting. Latest message I got from the gms logs are below. I checked the same error and it seems there is a problem with the images and the tags.

2024-04-11 19:17:52,585 [ThreadPoolTaskExecutor-1] WARN  c.l.m.b.k.DataHubUpgradeKafkaListener:116 - System version is not up to date: v0.13.1rc1-0. Waiting for datahub-upgrade to complete...```
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

The message indicates that the datahub-upgrade job has not completed and we would need look at its logs to figure out what went wrong. See https://datahubproject.io/docs/troubleshooting/quickstart/#how-can-i-confirm-if-all-docker-containers-are-running-as-expected-after-a-quickstart for more details.

4

datahub-upgrade container runs before the datahub-gms container and it’s status was exited.![attachment]({‘ID’: ‘F06TQ5VC2VD’, ‘EDITABLE’: False, ‘IS_EXTERNAL’: False, ‘USER_ID’: ‘U02JDQZ26G4’, ‘CREATED’: ‘2024-04-12 20:57:20+00:00’, ‘PERMALINK’: ‘Slack’, ‘EXTERNAL_TYPE’: ‘’, ‘TIMESTAMPS’: ‘2024-04-12 20:57:20+00:00’, ‘MODE’: ‘hosted’, ‘DISPLAY_AS_BOT’: False, ‘PRETTY_TYPE’: ‘PNG’, ‘NAME’: ‘image.png’, ‘IS_PUBLIC’: True, ‘PREVIEW_HIGHLIGHT’: None, ‘MIMETYPE’: ‘image/png’, ‘PERMALINK_PUBLIC’: ‘https://slack-files.com/TUMKD5EGJ-F06TQ5VC2VD-cc9d39fb97’, ‘FILETYPE’: ‘png’, ‘EDIT_LINK’: None, ‘URL_PRIVATE’: ‘Slack’, ‘HAS_RICH_PREVIEW’: False, ‘TITLE’: ‘image.png’, ‘IS_STARRED’: False, ‘PREVIEW_IS_TRUNCATED’: None, ‘URL_PRIVATE_DOWNLOAD’: ‘Slack’, ‘PREVIEW’: None, ‘PUBLIC_URL_SHARED’: False, ‘MESSAGE_TS’: ‘1712955453.390959’, ‘PARENT_MESSAGE_TS’: ‘1712866792.604269’, ‘MESSAGE_CHANNEL_ID’: ‘C029A3M079U’, ‘_FIVETRAN_DELETED’: False, ‘LINES_MORE’: None, ‘LINES’: None, ‘SIZE’: 138380, ‘_FIVETRAN_SYNCED’: ‘2024-04-14 12:55:06.485000+00:00’})

5

datahub-gms container checks the system version and finds a difference. If this is the real problem then datahub-upgrade container does not do what is expected to do or datahub-gms system check should be changed.

6

datahub-upgrade posts a message on a Kafka topic which datahub-gms reads to check that the upgrade was completed and that datahub-upgrade was for the same version. Same as before, we would need to see the logs to provide further guidance.

7

here are the logs for the containers![attachment]({‘ID’: ‘F06TQFG0763’, ‘EDITABLE’: True, ‘IS_EXTERNAL’: False, ‘USER_ID’: ‘U02JDQZ26G4’, ‘CREATED’: ‘2024-04-12 22:34:30+00:00’, ‘PERMALINK’: ‘Slack’, ‘EXTERNAL_TYPE’: ‘’, ‘TIMESTAMPS’: ‘2024-04-12 22:34:30+00:00’, ‘MODE’: ‘snippet’, ‘DISPLAY_AS_BOT’: False, ‘PRETTY_TYPE’: ‘Plain Text’, ‘NAME’: ‘datahub_gms_logs.txt’, ‘IS_PUBLIC’: True, ‘PREVIEW_HIGHLIGHT’: ‘

\n
\n
Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts
\n
2024-04-12 22:09:19,585 [main] INFO  o.s.web.context.ContextLoader:255 - Root WebApplicationContext: initialization started
\n
2024-04-12 22:09:21,818 [main] INFO  io.ebean:37 - ebean version: unknown
\n
2024-04-12 22:09:21,827 [main] WARN  io.avaje.config:219 - Please rename application.yml to application.yaml - Using yml suffix (rather than yaml) is deprecated.
\n
2024-04-12 22:09:21,831 [main] INFO  io.avaje.config:54 - Loaded properties from [resource:application.yml]
\n
\n
\n’, ‘MIMETYPE’: ‘text/plain’, ‘PERMALINK_PUBLIC’: ‘https://slack-files.com/TUMKD5EGJ-F06TQFG0763-3eed3aa48e’, ‘FILETYPE’: ‘text’, ‘EDIT_LINK’: ‘Slack’, ‘URL_PRIVATE’: ‘Slack’, ‘HAS_RICH_PREVIEW’: False, ‘TITLE’: ‘datahub_gms_logs.txt’, ‘IS_STARRED’: False, ‘PREVIEW_IS_TRUNCATED’: True, ‘URL_PRIVATE_DOWNLOAD’: ‘Slack’, ‘PREVIEW’: 'Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts\n2024-04-12 22:09:19,585 [main] INFO o.s.web.context.ContextLoader:255 - Root WebApplicationContext: initialization started\n2024-04-12 22:09:21,818 [main] INFO io.ebean:37 - ebean version: unknown\n2024-04-12 22:09:21,827 [main] WARN io.avaje.config:219 - Please rename application.yml to application.yaml - Using yml suffix (rather than yaml) is deprecated.\n2024-04-12 22:09:21,831 [main] INFO io.avaje.config:54 - Loaded properties from [resource:application.yml] ', ‘PUBLIC_URL_SHARED’: False, ‘MESSAGE_TS’: ‘1712961283.738769’, ‘PARENT_MESSAGE_TS’: ‘1712866792.604269’, ‘MESSAGE_CHANNEL_ID’: ‘C029A3M079U’, 'FIVETRAN_DELETED’: False, ‘LINES_MORE’: 1839, ‘LINES’: 1844, ‘SIZE’: 143895, 'FIVETRAN_SYNCED’: ‘2024-04-14 12:55:06.573000+00:00’})![attachment]({‘ID’: ‘F06UHMZTAE5’, ‘EDITABLE’: True, ‘IS_EXTERNAL’: False, ‘USER_ID’: ‘U02JDQZ26G4’, ‘CREATED’: ‘2024-04-12 22:34:32+00:00’, ‘PERMALINK’: ‘Slack’, ‘EXTERNAL_TYPE’: ‘’, ‘TIMESTAMPS’: ‘2024-04-12 22:34:32+00:00’, ‘MODE’: ‘snippet’, ‘DISPLAY_AS_BOT’: False, ‘PRETTY_TYPE’: ‘Plain Text’, ‘NAME’: ‘datahub_upgrade_logs.txt’, ‘IS_PUBLIC’: True, ‘PREVIEW_HIGHLIGHT’: '
\n
\n
\n
  .   ____          _            __ _ 
\n
 /\\ / ' __ _ () __  __ _ \ \ \ \
\n
( ( )\ | ' | '| | ' \/ _ | \\ \\ \\ \\</pre></div>\n<div><pre> \\\\/  ___)| |_)| | | | | || (_| |  ) ) ) )</pre></div>\n</div>\n</div>\n', 'MIMETYPE': 'text/plain', 'PERMALINK_PUBLIC': 'https://slack-files.com/TUMKD5EGJ-F06UHMZTAE5-e032bbed95', 'FILETYPE': 'text', 'EDIT_LINK': 'https://datahubspace.slack.com/files/U02JDQZ26G4/F06UHMZTAE5/datahub_upgrade_logs.txt/edit', 'URL_PRIVATE': 'https://files.slack.com/files-pri/TUMKD5EGJ-F06UHMZTAE5/datahub_upgrade_logs.txt', 'HAS_RICH_PREVIEW': False, 'TITLE': 'datahub_upgrade_logs.txt', 'IS_STARRED': False, 'PREVIEW_IS_TRUNCATED': True, 'URL_PRIVATE_DOWNLOAD': 'https://files.slack.com/files-pri/TUMKD5EGJ-F06UHMZTAE5/download/datahub_upgrade_logs.txt', 'PREVIEW': "\n  .   ____          _            __ _ _\n /\\\\ / ___'_ __ _ _(_)_ __  __ _ \\ \\ \\ \\\n( ( )\\___ | '_ | '_| | '_ \\/ _ | \ \ \ \\n \\/  _)| |)| | | | | || (| |  ) ) ) )", ‘PUBLIC_URL_SHARED’: False, ‘MESSAGE_TS’: ‘1712961283.738769’, ‘PARENT_MESSAGE_TS’: ‘1712866792.604269’, ‘MESSAGE_CHANNEL_ID’: ‘C029A3M079U’, ‘_FIVETRAN_DELETED’: False, ‘LINES_MORE’: 2390, ‘LINES’: 2395, ‘SIZE’: 755088, ‘_FIVETRAN_SYNCED’: ‘2024-04-14 12:55:06.573000+00:00’})

            


            

              
              
              
            


          

          

            

              



              
                  
                  
              8
              
            

            

              
The version of the datahub-upgrade job is v0.13.1-0 but GMS is running version v0.13.1rc1-0  - first make sure that the image tag for the gms deployment is correct. next, scale down the gms deployment to 0 replicas. Run deploy again.


2024-04-12 22:15:49,686 [ThreadPoolTaskExecutor-1] WARN  c.l.m.b.k.DataHubUpgradeKafkaListener:116 - System version is not up to date: v0.13.1rc1-0. Waiting for datahub-upgrade to complete...```