Troubleshooting `FileNotFoundError` for Missing `manifest.json` in DataHub Ingestion Process

ingestion
1

Original Slack Thread

<@U06TM7M40RK> Can you help with below error though we have files in place?
<class ‘FileNotFoundError’>: [Errno 2] No such file or directory: '/home/ssm-user/.local/lib/python3.10/site-packages/datahub/cli/manifest.json

2

It seems likely that the error you’re encountering is due to the manifest.json file not being found at the specified path. This type of error often occurs when the file path is incorrect or the file does not exist at the specified location. Here are a few steps you can take to troubleshoot and resolve this issue:

  1. Verify File Path: Double-check the file path to ensure it is correct. Make sure that the manifest.json file is indeed located in the /home/ssm-user/.local/lib/python3.10/site-packages/datahub/cli/ directory.

  2. Check File Permissions: Ensure that the user running the DataHub process has the necessary permissions to access the file. You can modify the permissions using the chmod command if needed.

  3. File Existence: Confirm that the manifest.json file actually exists in the specified directory. You can use the ls command to list the files in the directory and verify its presence.

  4. Environment Setup: If you are using a virtual environment, ensure that it is activated and that the file paths are correctly set up within that environment.

  5. Reinstallation: If the file is missing, consider reinstalling the DataHub CLI to ensure all necessary files are correctly installed. You can do this by running:

    python3 -m pip install --upgrade acryl-datahub

  6. Docker or Kubernetes: If you are running DataHub in a containerized environment, ensure that the file is correctly mounted or copied into the container.

If these steps do not resolve the issue, it might be helpful to provide more context or logs to further diagnose the problem. Would you like more detailed instructions or have any specific questions about the setup process?

Sources:

3 
Obtaining venv creation lock...
Acquired venv creation lock
venv is already set up
venv setup time = 0 sec
This version of datahub supports report-to functionality
+ exec datahub ingest run -c /tmp/datahub/ingest/af12d685-775c-4b24-a817-5a10ff1c598c/recipe.yml --report-to /tmp/datahub/logs/af12d685-775c-4b24-a817-5a10ff1c598c/artifacts/ingestion_report.json
[2025-03-26 13:13:00,686] INFO     {datahub.cli.ingest_cli:150} - DataHub CLI version: 1.0.0rc2
[2025-03-26 13:13:00,689] INFO     {datahub.ingestion.run.pipeline:256} - No sink configured, attempting to use the default datahub-rest sink.
[2025-03-26 13:13:00,700] INFO     {datahub.ingestion.run.pipeline:273} - Sink configured successfully. DataHubRestEmitter: configured to talk to <http://datahub-gms:8080>
[2025-03-26 13:13:00,928] INFO     {datahub.ingestion.source.state.stateful_ingestion_base:241} - Stateful ingestion will be automatically enabled, as datahub-rest sink is used or `datahub_api` is specified
[2025-03-26 13:13:00,936] INFO     {datahub.ingestion.run.pipeline:298} - Source configured successfully.
[2025-03-26 13:13:00,938] INFO     {datahub.cli.ingest_cli:131} - Starting metadata ingestion
[2025-03-26 13:13:00,938] ERROR    {datahub.ingestion.run.pipeline:741} - Ingestion pipeline threw an uncaught exception: [Errno 2] No such file or directory: '/home/ssm-user/.local/lib/python3.10/site-packages/datahub/cli/manifest.json'
Traceback (most recent call last):
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 466, in run
    for wu in itertools.islice(
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/source/state/stale_entity_removal_handler.py", line 72, in auto_stale_entity_removal
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/auto_work_units/auto_ensure_aspect_size.py", line 91, in ensure_aspect_size
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/auto_work_units/auto_dataset_properties_aspect.py", line 62, in auto_patch_last_modified
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 148, in auto_workunit_reporter
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 252, in auto_browse_path_v2
    for urn, batch in _batch_workunits_by_urn(stream):
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 508, in _batch_workunits_by_urn
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 415, in auto_fix_empty_field_paths
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 367, in auto_fix_duplicate_schema_field_paths
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 171, in auto_materialize_referenced_tags_terms
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 103, in auto_status_aspect
    for wu in stream:
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/source/dbt/dbt_common.py", line 956, in get_workunits_internal
    all_nodes, additional_custom_props = self.load_nodes()
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/source/dbt/dbt_core.py", line 575, in load_nodes
    ) = self.loadManifestAndCatalog()
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/source/dbt/dbt_core.py", line 513, in loadManifestAndCatalog
    dbt_manifest_json = self.load_file_as_json(
  File "/tmp/datahub/ingest/venv-dbt-365746d36fe52389/lib/python3.10/site-packages/datahub/ingestion/source/dbt/dbt_core.py", line 500, in load_file_as_json
    with open(uri) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ssm-user/.local/lib/python3.10/site-packages/datahub/cli/manifest.json'
[2025-03-26 13:13:00,945] INFO     {datahub.ingestion.reporting.file_reporter:54} - Wrote FAILURE report successfully to &lt;_io.TextIOWrapper name='/tmp/datahub/logs/af12d685-775c-4b24-a817-5a10ff1c598c/artifacts/ingestion_report.json' mode='w' encoding='UTF-8'&gt;
[2025-03-26 13:13:00,945] INFO     {datahub.cli.ingest_cli:144} - Finished metadata ingestion
Pipeline finished with at least 1 failures; produced 0 events in 0 seconds.
4

<@U06TM7M40RK> see above complete log file