Managing Different Data Pipes from Airflow Clusters in DataHub

ingestion
1

Original Slack Thread

Hi team~
I have two airflow clusters with the same Dag Id and different workflow configurations. When I do ingestion from both airflow clusters to datahub, all tasks are recognized as one data pipe because the Dag Id is the same. Is there any way to recognize them as different data pipes?

My environment is as follows

  1. Are you using UI or CLI for ingestion? UI + airflow plugin
  2. Which DataHub version are you using? 0.11.0
  3. What data source(s) are you integrating with DataHub? apache airflow
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

we can use cluster field of [datahub] configuration in airflow.cfg to different airflow instances. I believe the value to the field ‘cluster’ is going to be part URN’s of airflow entities
https://datahubproject.io/docs/lineage/airflow/#configuration

4

+1 on <@U0445MUD81W>’s answer

5

<@U0445MUD81W> <@UV14447EU>
Thanks bro