Troubleshooting DataHub Upgrade to 0.12.0: Connecting GMS and Solving Job Failure

troubleshoot
21

here’s an edited version of our values file https://gist.github.com/alexharv074/c30824bd9871ee2ffcd76f2002135fc7

22

That’s helpful to understand your values.yaml. If I take those values and render the helm template into manifests using datahub as the release name I get the following (see attached). One of the templates is clearly the system-update job that I would expect. The helm manifests look correct given those values. This means that there is something with your argocd setup that is not executing the job. The specific manifest in the file looks like this on line 934 in the attached file.

apiVersion: batch/v1
kind: Job
metadata:
  name: datahub-datahub-system-update-job
  labels:
    <http://helm.sh/chart|helm.sh/chart>: datahub-0.3.14
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: datahub
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: datahub
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: "0.11.0"
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  annotations:
    <http://helm.sh/hook|helm.sh/hook>: pre-install,pre-upgrade
    <http://helm.sh/hook-delete-policy|helm.sh/hook-delete-policy>: before-hook-creation
    <http://helm.sh/hook-weight|helm.sh/hook-weight>: "-4"
spec:
  template:
    spec:
      volumes:
      restartPolicy: Never
      securityContext:
        {}
      containers:
        - name: datahub-system-update-job
          image: "acryldata/datahub-upgrade:v0.12.0"
          imagePullPolicy: IfNotPresent
          args:
            - "-u"
            - "SystemUpdate"
          env:
            - name: DATAHUB_REVISION
              value: "1"
            - name: ENTITY_REGISTRY_CONFIG_PATH
              value: /datahub/datahub-gms/resources/entity-registry.yml
            - name: DATAHUB_GMS_HOST
              value: datahub-datahub-gms
            - name: DATAHUB_GMS_PORT
              value: "8080"
            - name: DATAHUB_MAE_CONSUMER_HOST
              value: datahub-datahub-mae-consumer
            - name: DATAHUB_MAE_CONSUMER_PORT
              value: "9091"
            - name: EBEAN_DATASOURCE_USERNAME
              value: "root"
            - name: EBEAN_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: "mysql-secrets"
                  key: "mysql-root-password"
            - name: EBEAN_DATASOURCE_HOST
              value: "prerequisites-mysql:3306"
            - name: EBEAN_DATASOURCE_URL
              value: "jdbc:<mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&amp;useSSL=true&amp;useUnicode=yes&amp;characterEncoding=UTF-8&amp;enabledTLSProtocols=TLSv1.2>"
            - name: EBEAN_DATASOURCE_DRIVER
              value: "com.mysql.jdbc.Driver"
            - name: KAFKA_BOOTSTRAP_SERVER
              value: "prerequisites-kafka:9092"
            - name: KAFKA_PRODUCER_COMPRESSION_TYPE
              value: "none"
            - name: KAFKA_PRODUCER_MAX_REQUEST_SIZE
              value: "5242880"
            - name: KAFKA_CONSUMER_MAX_PARTITION_FETCH_BYTES
              value: "5242880"
            - name: KAFKA_SCHEMAREGISTRY_URL
              value: <http://datahub-datahub-gms:8080/schema-registry/api/>
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch-master"
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: SKIP_ELASTICSEARCH_CHECK
              value: "false"
            - name: ELASTICSEARCH_INSECURE
              value: "false"
            - name: ELASTICSEARCH_USE_SSL
              value: "false"
            - name: GRAPH_SERVICE_IMPL
              value: elasticsearch
            - name: METADATA_CHANGE_EVENT_NAME
              value: MetadataChangeEvent_v4
            - name: FAILED_METADATA_CHANGE_EVENT_NAME
              value: FailedMetadataChangeEvent_v4
            - name: METADATA_AUDIT_EVENT_NAME
              value: MetadataAuditEvent_v4
            - name: METADATA_CHANGE_PROPOSAL_TOPIC_NAME
              value: MetadataChangeProposal_v1
            - name: FAILED_METADATA_CHANGE_PROPOSAL_TOPIC_NAME
              value: FailedMetadataChangeProposal_v1
            - name: METADATA_CHANGE_LOG_VERSIONED_TOPIC_NAME
              value: MetadataChangeLog_Versioned_v1
            - name: METADATA_CHANGE_LOG_TIMESERIES_TOPIC_NAME
              value: MetadataChangeLog_Timeseries_v1
            - name: DATAHUB_UPGRADE_HISTORY_TOPIC_NAME
              value: DataHubUpgradeHistory_v1
            - name: DATAHUB_ANALYTICS_ENABLED
              value: "true"
            - name: SCHEMA_REGISTRY_SYSTEM_UPDATE
              value: "true"
            - name: SPRING_KAFKA_PROPERTIES_AUTO_REGISTER_SCHEMAS
              value: "true"
            - name: SPRING_KAFKA_PROPERTIES_USE_LATEST_VERSION
              value: "true"
            - name: SCHEMA_REGISTRY_TYPE
              value: "INTERNAL"
            - name: ELASTICSEARCH_BUILD_INDICES_CLONE_INDICES
              value: "true"
            - name: ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX
              value: "true"
            - name: ELASTICSEARCH_INDEX_BUILDER_SETTINGS_REINDEX
              value: "true"
          securityContext:
            {}
          volumeMounts:
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 300m
              memory: 256Mi```![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F068DURK5SB/rendered.yaml?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
23

Thanks <@U03MF8MU5P0> very strange! We are investigating ArgoCD at our end to see if we can find an issue.