Hello,
I’ve just deployed the last datahub version and the ingestion through the UI doesn’t work.
Datahub Chart Version: 0.4.10```
When trying to debug, I found a possible error on the elasticsearch setup job. Is it normal to have `sed: /index/usage-event/datahub_usage_event: No such file or directory` ?
Also, when I ingest a source through the UI, it status get stuck on `Running` forever. But the deployment seems to be healthy. So I don't know what can be causing it:
*datahub-actions logs:*
```2024/05/18 14:01:44 Received 200 from <http://datacatalog-datahub-gms:8080/health>
No user action configurations found. Not starting user actions.
[2024-05-18 14:01:47,776] INFO {datahub_actions.cli.actions:80} - DataHub Actions version: unavailable (installed editable via git)
[2024-05-18 14:01:49,025] WARNING {datahub_actions.cli.actions:107} - Skipping pipeline datahub_slack_action as it is not enabled
[2024-05-18 14:01:49,036] WARNING {datahub_actions.cli.actions:107} - Skipping pipeline datahub_teams_action as it is not enabled
[2024-05-18 14:01:49,038] INFO {datahub_actions.cli.actions:123} - Action Pipeline with name 'ingestion_executor' is now running.```
*datahub-frontend:*
```13:59:46,816 |-INFO in ch.qos.logback.classic.util.ContextInitializer@38089a5a - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 534 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY
2024-05-18 13:59:49,141 [application-akka.actor.default-dispatcher-6] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2024-05-18 13:59:52,951 [main] WARN c.l.r.t.h.client.HttpClientFactory - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
2024-05-18 13:59:52,953 [main] WARN c.l.r.t.h.client.HttpClientFactory - No callback executor is provided to HttpClientFactory, using it's own call back executor.
2024-05-18 13:59:52,956 [main] WARN c.l.r.t.h.client.HttpClientFactory - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
2024-05-18 13:59:52,977 [main] INFO c.l.r.t.h.client.HttpClientFactory - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 ', http.protocolVersion=HTTP_1_1, usePipelineV2=false, requestTimeout=10000ms, streamingTimeout=-1ms
2024-05-18 13:59:54,467 [main] WARN auth.sso.SsoManager - Failed to get SSO settings due to exception, reusing previous settings
org.apache.http.conn.HttpHostConnectException: Connect to datacatalog-datahub-gms:8080 [datacatalog-datahub-gms/10.180.12.5] failed: Connection refused
...
2024-05-18 13:59:55,461 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka version: 5.5.15-ccs
2024-05-18 13:59:55,461 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka commitId: 42c2f4267b745647
2024-05-18 13:59:55,464 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1716040795459
2024-05-18 13:59:55,489 [main] INFO play.api.Play - Application started (Prod) (no global state)
2024-05-18 13:59:56,457 [main] INFO server.CustomAkkaHttpServer - Setting max header count to: 64
2024-05-18 13:59:56,718 [kafka-producer-network-thread | datahub-frontend] INFO org.apache.kafka.clients.Metadata - [Producer clientId=datahub-frontend] Cluster ID: rB1HQJbTRWGXvbByXm-5zw
2024-05-18 13:59:57,286 [main] INFO play.core.server.AkkaHttpServer - Listening for HTTP on /0.0.0.0:9002
2024-05-18 14:00:44,345 [proxyClient-akka.actor.default-dispatcher-5] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2024-05-18 14:17:52,957 [application-akka.actor.default-dispatcher-13] WARN p.api.mvc.DefaultJWTCookieDataCodec - decode: cookie has invalid signature! message = JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2024-05-18 14:17:52,958 [application-akka.actor.default-dispatcher-13] INFO p.api.mvc.DefaultJWTCookieDataCodec - The JWT signature in the cookie does not match the locally computed signature with the server. This usually indicates the browser has a leftover cookie from another Play application, so clearing cookies may resolve this error message.
2024-05-18 14:17:58,058 [application-akka.actor.default-dispatcher-6] WARN o.e.j.j.spi.PropertyFileLoginModule - Exception starting propertyUserStore /etc/datahub/plugins/frontend/auth/user.props
java.lang.IllegalStateException: Config does not exist: file:///etc/datahub/plugins/frontend/auth/user.props
at org.eclipse.jetty.security.PropertyUserStore.loadUsers(PropertyUserStore.java:219)
at org.eclipse.jetty.security.PropertyUserStore.doStart(PropertyUserStore.java:285)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.eclipse.jetty.jaas.spi.PropertyFileLoginModule.setupPropertyUserStore(PropertyFileLoginModule.java:105)
at org.eclipse.jetty.jaas.spi.PropertyFileLoginModule.initialize(PropertyFileLoginModule.java:56)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:745)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)```
*datahub-gms:*
```Unhealthy
Liveness probe failed: Get "<http://10.176.5.91:8080/health>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)```
```2024-05-18 14:03:46,656 [pool-16-thread-2] INFO c.l.m.entity.EntityServiceImpl:639 - Ingestion of aspects batch to database took 14 ms
2024-05-18 14:03:46,656 [pool-16-thread-2] INFO c.l.m.entity.EntityServiceImpl:1514 - Producing MCL for ingested aspect dataHubUpgradeResult, urn urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,659 [pool-16-thread-2] INFO c.l.m.b.s.RestoreDbtSiblingsIndices:124 - Successfully restored sibling aspects
2024-05-18 14:03:46,661 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:78 - Got MCL event key: urn:li:dataHubUpgrade:restore-dbt-siblings-indices, topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 2184, value size: 241, timestamp: 1716041026657
2024-05-18 14:03:46,662 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:102 - Invoking MCL hooks for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices, aspect name: dataHubUpgradeResult, entity type: dataHubUpgrade, change type: UPSERT
2024-05-18 14:03:46,662 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook SiblingAssociationHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,662 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook UpdateIndicesHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,663 [ThreadPoolTaskExecutor-1] INFO c.l.m.s.e.update.ESBulkProcessor:82 - Added request id: gR5PC881t9xI9pLa+gcF9Q==, operation type: UPDATE, index: datahub-orus__system_metadata_service_v1
2024-05-18 14:03:46,663 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook IngestionSchedulerHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook EntityChangeEventGeneratorHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook FormAssignmentHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:115 - Invoking MCL hook IncidentsSummaryHook for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:46,664 [ThreadPoolTaskExecutor-1] INFO c.l.m.k.MetadataChangeLogProcessor:133 - Successfully completed MCL hooks for urn: urn:li:dataHubUpgrade:restore-dbt-siblings-indices
2024-05-18 14:03:47,199 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:61 - Successfully fed bulk request 10. Number of events: 3 Took time ms: 10```
*Ingestion Logs from UI*:
```Building wheels for collected packages: avro
Building wheel for avro (pyproject.toml): started
Building wheel for avro (pyproject.toml): finished with status 'done'
Created wheel for avro: filename=avro-1.11.3-py2.py3-none-any.whl size=123913 sha256=01b196d8c3931f06ed32753c8fccf0c7992e172bd969b1d9bd0fee2e4a26ab61
Stored in directory: /datahub-ingestion/.cache/pip/wheels/1d/f6/41/0e0399396af07060e64d4e32c8bd259b48b98a4a114df31294
Successfully built avro
Installing collected packages: webencodings, wcwidth, vininfo, pytz, python-stdnum, pure-eval, ptyprocess, phonenumbers, makefun, ipython-genutils, ipaddress, ijson, fastjsonschema, cymem, confluent-kafka, click-spinner, cached-property, zipp, wrapt, widgetsnbextension, websocket-client, wasabi, urllib3, uc-micro-py, tzlocal, tzdata, typing-extensions, traitlets, tqdm, tornado, toolz, toml, tinycss2, termcolor, tabulate, sqlparse, spacy-loggers, spacy-legacy, soupsieve, sniffio, smart-open, six, send2trash, ruamel.yaml, rstr, rpds-py, pyzmq, PyYAML, pyparsing, pygments, pycparser, pycountry, psycopg2-binary, psutil, prompt-toolkit, prometheus-client, pexpect, pathlib-abc, parso, pandocfilters, packaging, numpy, nest-asyncio, mypy-extensions, murmurhash, multidict, mistune, mdurl, MarkupSafe, marisa-trie, jupyterlab-widgets, jupyterlab-pygments, jsonref, jsonpointer, importlib-resources, idna, humanfriendly, greenlet, frozenlist, fastavro, expandvars, executing, exceptiongroup, entrypoints, defusedxml, decorator, debugpy, colorama, click, charset-normalizer, certifi, catalogue, avro, attrs, async-timeout, acryl-sqlglot, yarl, typing-inspect, typer, terminado, srsly, sqlalchemy, sentry-sdk, scipy, schwifty, requests, referencing, python-utils, python-dateutil, pydantic, preshed, matplotlib-inline, marshmallow, markdown-it-py, linkify-it-py, language-data, jupyter-core, jsonpatch, jinja2, jedi, importlib-metadata, Deprecated, comm, click-default-group, cffi, blis, bleach, beautifulsoup4, avro-gen3, asttokens, anyio, aiosignal, stack-data, rich, requests-file, progressbar2, pathy, pandas, mixpanel, mdit-py-plugins, langcodes, jupyter-client, jsonschema-specifications, GeoAlchemy2, docker, cryptography, confection, argon2-cffi-bindings, aiohttp, thinc, jsonschema, Ipython, argon2-cffi, textual, spacy, nbformat, ipywidgets, ipykernel, altair, acryl-datahub, nbclient, memray, acryl-datahub-classify, nbconvert, jupyter-server, notebook-shim, nbclassic, notebook, great-expectations```
Make sure your message includes the following information if relevant, so we can help more effectively!