Understanding the Difference Between `acryl-spark-lineage` and `datahub-spark-lineage`

ingestion
1

Original Slack Thread

Hello!

I’m a bit confused regarding spark openlineage plugin. What is the difference:
https://central.sonatype.com/artifact/io.acryl/acryl-spark-lineage/versions and this
https://central.sonatype.com/artifact/io.acryl/datahub-spark-lineage/versions

is it kind of rebranding and one of these is outdated (datahub-spark-lineage)?

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

The confusion between acryl-spark-lineage and datahub-spark-lineage stems from a rebranding and update in the DataHub project. Here’s a breakdown:

  1. Rebranding and Update: The datahub-spark-lineage has been rebranded to acryl-spark-lineage. This change reflects a broader scope and potential rebranding of the project. The acryl-spark-lineage is the newer version and includes more features and improvements.

  2. Current Recommendation: According to a <Slack conversation>, it is recommended to use the acryl-spark-lineage as it is based on OpenLineage and offers more features and reliability compared to the older datahub-spark-lineage.

  3. Deprecation of Older Version: The datahub-spark-lineage is not maintained anymore and will be replaced by the new acryl-spark-lineage as mentioned in the <Issues · datahub-project/datahub · GitHub issue>.

Would you like more details on the features or setup of the acryl-spark-lineage?

Sources:

4

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><