How to Add Column-Level Lineage and Store it in MySQL with Sample Data

all-things-deployment
1

Original Slack Thread

Due to the inability to add column-level lineage on the webUI, only table-level lineage can be added. How can I add column-level lineage?Therefore, I would like to know how column-level lineage is stored in MySQL. I noticed that table-level lineage is stored as “aspect:upstreamLineage” in the table metadata_aspect_v2. Can someone provide an answer? It would be great if a sample data of column-level lineage can be provided, or even a sample image. Just like mine::blush:

attachment
attachment1875×918 605 KB

2

<@U05CJD391ND> might be able to speak to this!

3

I have referred to the official website and added column lineage using a Python script, but the official website https://datahubproject.io/docs/api/tutorials/lineage/#add-column-level-lineage does not provide instructions on how to add column-level lineage between multiple columns in two tables using Python.

For example:
table1 table2
id--------->id1
name------->name1

4

Hi <@U05PB6VG9JA>, I’m sorry you’re having trouble with column-level lineage. <@U04N9PYJBEW> do you think you could help out here?

5

Each FineGrainedLineage object in the docs you linked creates a column-level lineage edge. Thus, to accomplish your scenario, you could do:

  FineGrainedLineage(upstreamType=FIELD_SET, upstreams=[fldUrn("table1", "id")], downstreamType=FIELD, downstreams=[fldUrn("table2", "id1")]),
  FineGrainedLineage(upstreamType=FIELD_SET, upstreams=[fldUrn("table1", "name")], downstreamType=FIELD, downstreams=[fldUrn("table2", "name1")]),
]```