Hi - I am working to troubleshoot why my DataHub deployment (version 0.12.0) has:
- Multiple copies of the same data entities
- Working & Non-working lineage
Background - My DataHub version ingests from both BQ and DBT (recent addition). Since adding DBT, I have had to add
convert_urns_to_lowercase
to my BQ recipe to ensure table urns of the same data entity align. - Some ARDs(Views) are complex and are generated from multiple SQL scripts which are executed via a dataflow job. The data from these Views appears to be correct in BigQuery and DataHub, only the lineage fails (i.e. unavailable or ends unexpectedly).
Problem(s) - Multiple entities in DataHub: old BQ uppercase urn, new BQ&DBT lowercase urn, and/or new BQ lowercase urns.
- Lineage works for simple ARDs, but fails as the work to create to an ARD gets increasingly more complex.
Question - How can I align DBT and BQ tables/views to use uppercase urns only? I would rather use the uppercase option as this is more reflective of the underlying data.
- How can I ensure lineage works for all my tables/views? Does the BQ plugin struggle to read lineage at a certain level of complexity?
I am new to posting in the DataHub slack channel, so I hope this makes sense. I found both issues as part of the same investigation, so I am unsure if they are separate or related. I would appreciate any guidance that could help me troubleshoot these issues!