Hi. In the new browsing Experience I see that Airflow Pipelines got a new dimension - some ‘Default’ container. But how do I create more of such containers and divide pipelines into them?
I have several Airflow instances connected to Datahub, each in its own ingest recipe. Before I couldn’t find a way do separete them in UI, but maybe this new functionality can help me?
<@U01GCJKA8P9> might be able to speak to this!
hey <@U03LYB2ESJ0>! yes I think you can indeed use this new functionality for your use case. what’s going on here is we have a new browsePathsV2 aspect that we will use to show these folders underneath the platform - right now all of your airflow entities have a browsePathsV2 aspect that looks like this: {"path":[{"id":"Default"}]} - you have a few options here. (1) I would upgrade your ingestion version to get better browse paths here and (2) I believe if you use a data-platform-instance that is different on each of your ingestion recipes (can define these on the recipe) then we should create a top level path that splits between them
also here’s a doc giving more details on this new aspect as well as how you could generate custom ones: https://datahubproject.io/docs/browsev2/browse-paths-v2/
<@U03BEML16LB> (1) We are now on 10.5, do I need further upgrade to get better browse paths?
(2) but Airflow connection is push-based, not pull-based, so there’s no ingestione recipe in Datahub for it, but I believe there is a Datahub plugin installed on Airflow’s side. I’m not very familiar with this plugin, can those paths be set there the same way?
ahh okay sorry i totally forgot airflow was different than most other ingestion sources. Let me tag <@UV14447EU> our resident expert in airflow to get his take. There’s a chance we may not be generating these aspects for airflow which makes sense why you’d be seeing this “Default” folder
<@UV14447EU> do you have any advice on this?
We don’t generate browse path v2 from the Airflow plugin at the moment
<@UV14447EU> is it in the roadmap? or cah we create them on our own using some transformer/mutation/Python SDK?
I need to look after
<@UV14447EU> hi. Did you manage to check of it’s possible to specify Airflow containers here?