Investigating Spark Ingestion on Databricks with Datahub Lineage Issues

getting-started
21

same logs as before

22

this is my query planattachment

23

my code does: spark.table(table).filter(condition).select(col1, col2).write.mode(‘overwrite’).format(‘parquet’).save(path)

24

<@UV14447EU> It appeared a different task now!!! Niiicee!! But with no lineage to hive yet, do you know what it could be? the task is linked by table name? path?attachment

25

attachment
attachment1111×372 21.7 KB
attachment
attachment1109×509 41.1 KB

26

Parquet appears task without Lineage, Delta does not appear task

Parquet: InsertHadoopFsRelationCommand

27

Delta: SaveIntoDataSourceCommand

28

maybe I should include platform instance? i didn’t understand what it is

29

Path on task appears without dbfs:/ and on hive it appears with dbfs:/ scheme on location

30

Update:

Lineage worked for downstream hive tables!!! I had to write it as parquet and use saveAsTable method from Spark.

I’ll try more to get it done for delta tables. Just one question, spark integration ingest upstream hive tables too? <@UV14447EU>