Troubleshooting DataHub Ingestion for DBT Core

ingestion
1

Original Slack Thread

Hello everyone
I’m doing the POC on datahub
I trying to ingestion for DBT core in local
I’m facing issue like file not there

Attaching the log below

I created the manifest.json and catalog.json and placed in the locally

source:
    type: dbt
    config:
        manifest_path: /Users/arunkumar/Documents/Datatool/data_catalog/target/manifest.json
        target_platform: snowflake
        catalog_path: /Users/arunkumar/Documents/Datatool/data_catalog/target/catalog.json```

I saw you have helped some users regarding this issue in past
<@U02G4B6ADL6>![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06JH9KLYPQ/dbt.log?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

1.UI ingestion
2.DataHub CLI version: 0.12.1.5
3.DBT

4

Hey Arun, Could you please confirm if this file exists.

/Users/arunkumar/Documents/Datatool/data_catalog/target/manifest.json⁣

5

the file exist in my localattachment

6

But I read above the things in this channel
they saying we need to add the things in docker

7

Hey Arun, if you are using managed(UI) ingestion on your locally deployed datahub, you would need to have the file at that location in datahub-actions docker container. If using CLI ingestion on your machine, then it would suffice to have it at that path on your machine.

8

how to do the cli ingestion
Can you share sample things for dbt

9

https://datahubproject.io/docs/metadata-ingestion#basic-usage-of-cli-for-ingestion
basically create a recipe yaml file on your local and run it using datahub ingest -c &lt;recipe.yml&gt;

10

I created one file dbt.dhub.yml

  type: "dbt"
  config:
    manifest_path: "/Users/arunkumar/Documents/Datatool/data_catalog/target/manifest.json"
    catalog_path: "/Users/arunkumar/Documents/Datatool/data_catalog/target/catalog.json"

    target_platform: "snowflake"```
Attaching the error

```(data-catalog-py3.12) arunkumar@arunkumars-air data_catalog % datahub ingest -c dbt..dhub.yml
[2024-02-14 12:35:50,479] INFO     {datahub.cli.ingest_cli:147} - DataHub CLI version: 0.12.1.5
[2024-02-14 12:35:50,883] ERROR    {datahub.entrypoints:201} - Command failed: Cannot open config file dbt..dhub.yml
Traceback (most recent call last):```
11

there is a typo in command, please check.

12

Ohh Sorry

13

Its alright

14

Thanks for the support <@U02G4B6ADL6> <@U06DLBKN7HV>

15

I’m going to start the POC for airflow
Is there any documentation
How to test in local

16

How does your sink config look like ?

Here is more info on that - https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub#quickstart-recipe
Usually server is <http://localhost:8080> if you deployed datahub using quickstart.

(Btw, you could also copy your dbt files to datahub actions container manually to make this work in UI ingestion and not worry about sink configs)

Also <Slack quick check> may help.

17

Sure
Now I could see the things
Once I restarted the server

18

> I’m going to start the POC for airflow
> Is there any documentation
> How to test in local
https://datahubproject.io/docs/lineage/airflow/ . If you have any questions on airflow, please start a new thread so that others familiar with airflow can help.

19

Sure thanks

20

Is there any thing I need to see the lineage with looker and snowflake