I want to acheive column level lineage using Pyspark script in Datahub. Created Pyspark script to read data from mysql server (datahub docker img) using JDBC driver, performed some transformation and load it in the same location. I also ingested Mysql server in datahub Ref: https://datahubproject.io/docs/api/tutorials/lineage/#add-lineage (Add Column-level Lineage). My Task is reflecting correctly on spark platform in datahub but unable to achieve column level lineage. can some one please guide me on the same?
version:
datahub:v0.12.1
spark:v3.5.0
The Airflow plugin can only capture table/column level lineage from SQL queries (where it supports the operator), but on Pyspark, you will need the new spark plugin to capture it as there. To get the column-level lineage, the query execution context from Spark is needed.
If you use the spark SQL operator with pure SQL, I can see there is a chance it works.