Troubleshooting AWS DataHub Installation Issues with Docker and Elasticsearch Setup

troubleshoot
1

Original Slack Thread

Hello Team.
I’m in the process of setting up a datahub in an AWS environment.

At first, it was easy to install and operate with the datahub docker quickstart command in the AWS EC2 environment.
Within a few days, I got a message that said, “An unknown error occurred. (code 500)” error message and the ingested data is being deleted repeatedly.
( I’ve tried datahub versions 0.10.1 to 0.12.0 and it’s the same )
There is no proper GMS log, and judging from the log information in each Docker, I think the Elasticsearch service is shutting down abnormally.

I’m getting a code 500 error quite often, so I’m looking at https://datahubproject.io/docs/deploy/aws documentation and trying to set up a datahub.

The current status is that you have completed the AWS EKS setup and have completed the authentication process in AWS Certificate Manager.

But after modifying the values.yaml file,
helm upgrade --install datahub datahub/datahub --values values.yaml --debug
When I run the command, I get the following message and an error with timeout and the installation cannot continue.

history.go:56: [debug] getting history for release datahub
upgrade.go:153: [debug] preparing upgrade for datahub
upgrade.go:161: [debug] performing update for datahub
upgrade.go:354: [debug] creating upgraded release for datahub
client.go:486: [debug] Starting delete for "datahub-elasticsearch-setup-job" Job
client.go:142: [debug] creating 1 resource(s)
client.go:712: [debug] Watching for changes to Job datahub-elasticsearch-setup-job with timeout of 5m0s
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: ADDED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 1, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 1, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 2, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:779: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
upgrade.go:466: [debug] warning: Upgrade "datahub" failed: pre-upgrade hooks failed: 1 error occurred:
        * timed out waiting for the condition


Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
        * timed out waiting for the condition


helm.go:84: [debug] pre-upgrade hooks failed: 1 error occurred:
        * timed out waiting for the condition


UPGRADE FAILED
main.newUpgradeCmd.func2
        <http://helm.sh/helm/v3/cmd/helm/upgrade.go:229|helm.sh/helm/v3/cmd/helm/upgrade.go:229>
<http://github.com/spf13/cobra.(*Command).execute|github.com/spf13/cobra.(*Command).execute>
        <http://github.com/spf13/cobra@v1.7.0/command.go:940|github.com/spf13/cobra@v1.7.0/command.go:940>
<http://github.com/spf13/cobra.(*Command).ExecuteC|github.com/spf13/cobra.(*Command).ExecuteC>
        <http://github.com/spf13/cobra@v1.7.0/command.go:1068|github.com/spf13/cobra@v1.7.0/command.go:1068>
<http://github.com/spf13/cobra.(*Command).Execute|github.com/spf13/cobra.(*Command).Execute>
        <http://github.com/spf13/cobra@v1.7.0/command.go:992|github.com/spf13/cobra@v1.7.0/command.go:992>
main.main
        <http://helm.sh/helm/v3/cmd/helm/helm.go:83|helm.sh/helm/v3/cmd/helm/helm.go:83>
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1598```
After this result, pods looks like this:
```&gt; kubectl get pods
NAME                                    READY   STATUS   RESTARTS   AGE
datahub-elasticsearch-setup-job-bmkzh   0/1     Error    0          7m9s
datahub-elasticsearch-setup-job-r8688   0/1     Error    0          9m19s
datahub-elasticsearch-setup-job-wqr4g   0/1     Error    0          4m49s
datahub-elasticsearch-setup-job-zmnl7   0/1     Error    0          2m9s```
I would like to know how I can solve it.

And here, too, seems to be a problem with Elasticsearch. How do I use AWS opensearch instead of elasticsearch?![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F069728NWVC/image.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

What about the logs of the elasticsearch setup jobs? <@U04N6530BE0> any luck there?

4

When I checked the requested logs of the elasticsearch setup jobs, it seemed that the AWS environment was not able to connect to elasticsearch, so I am checking the AWS route and VPC environment.


2023/12/12 23:35:46 Waiting for: <http://elasticsearch-master:9200>
2023/12/12 23:35:56 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:52192-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:36:12 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:58441-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:36:23 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:54354-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:36:39 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:56061-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:36:55 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:43148-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:37:06 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:54480-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:37:22 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:53978-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:37:38 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp: lookup elasticsearch-master on 172.20.0.10:53: read udp [<http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>]:58423-&gt;172.20.0.10:53: read: connection refused. Sleeping 1s
2023/12/12 23:37:46 Timeout after 2m0s waiting on dependencies to become available: [<http://elasticsearch-master:9200>]
2023/12/12 23:37:46 Command exited with error: exit status 1```
If I use AWS Opensearch instead of Elasticsearch, isn't the elasticsearch setup job unnecessary?
Even if you modify the global: elasticsearch host entry in the value.yaml file to the opensearch arn address and run the command.
 &gt; helm upgrade --install datahub datahub/datahub --values values.yaml --debug
elasticsearch_setup_job the same error is occurring.