Troubleshooting Datahub Deployment with Kubernetes in Azure AKS

all-things-deployment
1

Original Slack Thread

Hi,

We are facing some problems when trying to deploy Datahub with Kubernetes in Azure Kubernetes Service (AKS). If I do a clean helm install of the prerequisites and the Datahub charts as in the tutorial

https://datahubproject.io/docs/deploy/kubernetes/ ,

I am able to install the prerequisites but the installation of Datahub keeps failing. The culprit is the datahub-datahub-system-update-job which keeps on failing.

If I modify the prerequisites and the datahub values.yaml as suggested here:

https://github.com/acryldata/datahub-helm/issues/347

This is:
“”"
Filling values in prerequisites:

  enabled: true
  cp-schema-registry:
    enabled: true```
and values in datahub:
```kafka:
  schemaregistry:
    type: KAFKA
    url: "<http://datahub-datahub-gms:8080/schema-registry/api/>"```
"""

I am able to helm install the prerequisites without issues. I am also able to helm install datahub. However, in the front-end I don't have an "Ingestion" tab. If I write "http:my-datahub.domain/ingestion" I get an error "Failed to load ingestion sources! An unexpected error occurred" (see image). If I try to ingest some data, I get another error: "Unauthorized to perform this action. Please contact your DataHub administrator (code 403)." (see image).

Does anyone have any clue on how to fix these errors?

Any tips would be greatly appreciated!![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F05QWDZKL49/screenshot_from_2023-08-31_23-01-27.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F05R7GA2PH6/screenshot_from_2023-08-31_23-01-44.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
2

<@U05CJD391ND> might be able to speak to this!

3

Hey <@U05PTUSKN4E> - the datahub upgrade job needs to finish successfully to bring up datahub in a good state. Have you looked at the logs for datahub-upgrade to see why it’s failing? I’d suspect that may be the first place we would want to start taking a look. After that, the unauthorized issue may be the user you’re logging in with may not have been granted the right permissions. Are you logged into an account that has the right permissions? Thinking about the MANAGE_INGESTION permission.

4

Hi <@U05CJD391ND>,

Thanks a lot for your reply. As mentioned, when I do a helm install using the latest helm charts, without doing any modification to the values.yaml the datahub-datahub-system-update-job keeps failing after several attempts. The pod’s logs with the error look like the attached file:

update-job-logs.txt
Now, in this github issue:

https://github.com/acryldata/datahub-helm/issues/347

I think they find the same error when installing with the helm chart 0.2.181 (one version ealier). They suggest the following modification to the values.yaml as a workaround:

“”"
Same problem.
Filling values in prerequisites:

  enabled: true
  cp-schema-registry:
    enabled: true```
and values in datahub:
```kafka:
  schemaregistry:
    type: KAFKA
    url: "<http://prerequisites-cp-schema-registry:8081>"```
&gt; Still does not work, and same error. So <https://datahubproject.io/docs/deploy/kubernetes/|documentation> is incomplete and not enough for deploy on kubernetes.
For me exactly this config worked and the switch between the registries worked without an recognizable issue so far.
"""

When applying these suggestions, we are able to deploy to AKS without errors. We would like the implications of modifying kafka's schemaregistry from type INTERNAL to type Kafka, as well as of enabling the cp-schema-registry in the prerequisites...![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F05QP1NDRT8/update-job-logs.txt?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)