Troubleshooting Datahub Integration with Airflow Plugin Installation

ingestion
1

Original Slack Thread

Hi All
Currently I’m doing the POC on datahub

For that I trying to test the airflow from local as per the documentation
https://datahubproject.io/docs/lineage/airflow/

I updated the airflow.cfg file with needed changes

[core]
lazy_load_plugins = False

[datahub]
enabled = true
conn_id = datahub_rest_default # or datahub_kafka_default

etc.

later I created the new connection on the airflow for Datahub

In the Plugin area I could not see anything any DAG to run

I installed below things too

pip install 'acryl-datahub-airflow-plugin[plugin-v1]'
pip install 'acryl-datahub-airflow-plugin[plugin-v1,datahub-kafka]'```

<@U02G4B6ADL6>
Can you help me here
Is there any thing I need to here![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06KGTSQVU2/screenshot_2024-02-19_at_4.57.09_pm.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06L5MNSQ64/screenshot_2024-02-19_at_4.58.34_pm.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06KGR4LZH9/screenshot_2024-02-19_at_5.00.58_pm.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

1.UI ingestion
2.DataHub CLI version: 0.12.1.5
3. airflow

<@U06DLBKN7HV> <@U02G4B6ADL6>

4

Hey Arun, could you please share which airflow version you are using?

5

<@U06DLBKN7HV>
2.3.2

6

I think the plugin should show up on the plugin page. Are you sure you installed the plugin into the same python environment where Airflow runs and also you restarted Airflow?

7

Actually I trying to connect the datahub from local to airflow production
I restarted the airflow once I done all the changes

I could not find any plugin in the plugin page

8

have you installed these plugins in your production Airflow nodes ?

pip install 'acryl-datahub-airflow-plugin[plugin-v1,datahub-kafka]'```