Troubleshooting Datahub Ingestion Issue with UI Deployment and Elasticsearch Setup

Original Slack Thread

I’ve just deployed the last datahub version and the ingestion through the UI doesn’t work.

Datahub Chart Version: 0.4.10```
When trying to debug, I found a possible error on the elasticsearch setup job. Is it normal to have `sed: /index/usage-event/datahub_usage_event: No such file or directory` ?
Also, when I ingest a source through the UI, it status get stuck on `Running` forever. But the deployment seems to be healthy. So I don't know what can be causing it:

*datahub-actions logs:*
```2024/05/18 14:01:44 Received 200 from <http://datacatalog-datahub-gms:8080/health>
No user action configurations found. Not starting user actions.
[2024-05-18 14:01:47,776] INFO     {datahub_actions.cli.actions:80} - DataHub Actions version: unavailable (installed editable via git)
[2024-05-18 14:01:49,025] WARNING  {datahub_actions.cli.actions:107} - Skipping pipeline datahub_slack_action as it is not enabled
[2024-05-18 14:01:49,036] WARNING  {datahub_actions.cli.actions:107} - Skipping pipeline datahub_teams_action as it is not enabled
[2024-05-18 14:01:49,038] INFO     {datahub_actions.cli.actions:123} - Action Pipeline with name 'ingestion_executor' is now running.```
```13:59:46,816 |-INFO in ch.qos.logback.classic.util.ContextInitializer@38089a5a - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 534 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY
2024-05-18 13:59:49,141 [] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2024-05-18 13:59:52,951 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
2024-05-18 13:59:52,953 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No callback executor is provided to HttpClientFactory, using it's own call back executor.
2024-05-18 13:59:52,956 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
2024-05-18 13:59:52,977 [main] INFO  c.l.r.t.h.client.HttpClientFactory - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 ', http.protocolVersion=HTTP_1_1, usePipelineV2=false, requestTimeout=10000ms, streamingTimeout=-1ms
2024-05-18 13:59:54,467 [main] WARN  auth.sso.SsoManager - Failed to get SSO settings due to exception, reusing previous settings
org.apache.http.conn.HttpHostConnectException: Connect to datacatalog-datahub-gms:8080 [datacatalog-datahub-gms/] failed: Connection refused
2024-05-18 13:59:55,461 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 5.5.15-ccs
2024-05-18 13:59:55,461 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId: 42c2f4267b745647
2024-05-18 13:59:55,464 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1716040795459
2024-05-18 13:59:55,489 [main] INFO  play.api.Play - Application started (Prod) (no global state)
2024-05-18 13:59:56,457 [main] INFO  server.CustomAkkaHttpServer - Setting max header count to: 64
2024-05-18 13:59:56,718 [kafka-producer-network-thread | datahub-frontend] INFO  org.apache.kafka.clients.Metadata - [Producer clientId=datahub-frontend] Cluster ID: rB1HQJbTRWGXvbByXm-5zw
2024-05-18 13:59:57,286 [main] INFO  play.core.server.AkkaHttpServer - Listening for HTTP on /
2024-05-18 14:00:44,345 [] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started 
2024-05-18 14:17:52,957 [] WARN  p.api.mvc.DefaultJWTCookieDataCodec - decode: cookie has invalid signature! message = JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2024-05-18 14:17:52,958 [] INFO  p.api.mvc.DefaultJWTCookieDataCodec - The JWT signature in the cookie does not match the locally computed signature with the server. This usually indicates the browser has a leftover cookie from another Play application, so clearing cookies may resolve this error message.
2024-05-18 14:17:58,058 [] WARN  o.e.j.j.spi.PropertyFileLoginModule - Exception starting propertyUserStore /etc/datahub/plugins/frontend/auth/user.props
java.lang.IllegalStateException: Config does not exist: file:///etc/datahub/plugins/frontend/auth/user.props
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(
	at org.eclipse.jetty.jaas.spi.PropertyFileLoginModule.setupPropertyUserStore(
	at org.eclipse.jetty.jaas.spi.PropertyFileLoginModule.initialize(
	at java.base/
	at java.base/$
	at java.base/$
	at java.base/
	at java.base/```
Liveness probe failed: Get "<>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)```
```2024-05-18 14:03:46,656 [pool-16-thread-2] INFO  c.l.m.entity.EntityServiceImpl:639 - Ingestion of aspects batch to database took 14 ms
2024-05-18 14:03:46,656 [pool-16-thread-2] INFO  c.l.m.entity.EntityServiceImpl:1514 - Producing MCL for ingested aspect dataHubUpgradeResult, urn urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,659 [pool-16-thread-2] INFO  c.l.m.b.s.RestoreDbtSiblingsIndices:124 - Successfully restored sibling aspects
2024-05-18 14:03:46,661 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:78 - Got MCL event key: urn:li:dataHubUpgrade:restore-dbt-siblings-indices, topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 2184, value size: 241, timestamp: 1716041026657
2024-05-18 14:03:46,662 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:102 - Invoking MCL hooks for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices, aspect name: dataHubUpgradeResult, entity type: dataHubUpgrade, change type: UPSERT
2024-05-18 14:03:46,662 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook SiblingAssociationHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,662 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook UpdateIndicesHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,663 [ThreadPoolTaskExecutor-1] INFO  c.l.m.s.e.update.ESBulkProcessor:82 - Added request id: gR5PC881t9xI9pLa+gcF9Q==, operation type: UPDATE, index: datahub-orus__system_metadata_service_v1
2024-05-18 14:03:46,663 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook IngestionSchedulerHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook EntityChangeEventGeneratorHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook FormAssignmentHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook IncidentsSummaryHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:133 - Successfully completed MCL hooks for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:47,199 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:61 - Successfully fed bulk request 10. Number of events: 3 Took time ms: 10```
*Ingestion Logs from UI*:
```Building wheels for collected packages: avro
  Building wheel for avro (pyproject.toml): started
  Building wheel for avro (pyproject.toml): finished with status 'done'
  Created wheel for avro: filename=avro-1.11.3-py2.py3-none-any.whl size=123913 sha256=01b196d8c3931f06ed32753c8fccf0c7992e172bd969b1d9bd0fee2e4a26ab61
  Stored in directory: /datahub-ingestion/.cache/pip/wheels/1d/f6/41/0e0399396af07060e64d4e32c8bd259b48b98a4a114df31294
Successfully built avro
Installing collected packages: webencodings, wcwidth, vininfo, pytz, python-stdnum, pure-eval, ptyprocess, phonenumbers, makefun, ipython-genutils, ipaddress, ijson, fastjsonschema, cymem, confluent-kafka, click-spinner, cached-property, zipp, wrapt, widgetsnbextension, websocket-client, wasabi, urllib3, uc-micro-py, tzlocal, tzdata, typing-extensions, traitlets, tqdm, tornado, toolz, toml, tinycss2, termcolor, tabulate, sqlparse, spacy-loggers, spacy-legacy, soupsieve, sniffio, smart-open, six, send2trash, ruamel.yaml, rstr, rpds-py, pyzmq, PyYAML, pyparsing, pygments, pycparser, pycountry, psycopg2-binary, psutil, prompt-toolkit, prometheus-client, pexpect, pathlib-abc, parso, pandocfilters, packaging, numpy, nest-asyncio, mypy-extensions, murmurhash, multidict, mistune, mdurl, MarkupSafe, marisa-trie, jupyterlab-widgets, jupyterlab-pygments, jsonref, jsonpointer, importlib-resources, idna, humanfriendly, greenlet, frozenlist, fastavro, expandvars, executing, exceptiongroup, entrypoints, defusedxml, decorator, debugpy, colorama, click, charset-normalizer, certifi, catalogue, avro, attrs, async-timeout, acryl-sqlglot, yarl, typing-inspect, typer, terminado, srsly, sqlalchemy, sentry-sdk, scipy, schwifty, requests, referencing, python-utils, python-dateutil, pydantic, preshed, matplotlib-inline, marshmallow, markdown-it-py, linkify-it-py, language-data, jupyter-core, jsonpatch, jinja2, jedi, importlib-metadata, Deprecated, comm, click-default-group, cffi, blis, bleach, beautifulsoup4, avro-gen3, asttokens, anyio, aiosignal, stack-data, rich, requests-file, progressbar2, pathy, pandas, mixpanel, mdit-py-plugins, langcodes, jupyter-client, jsonschema-specifications, GeoAlchemy2, docker, cryptography, confection, argon2-cffi-bindings, aiohttp, thinc, jsonschema, Ipython, argon2-cffi, textual, spacy, nbformat, ipywidgets, ipykernel, altair, acryl-datahub, nbclient, memray, acryl-datahub-classify, nbconvert, jupyter-server, notebook-shim, nbclassic, notebook, great-expectations```

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!