Dealing with Deprecated OpenLineage Package in Datahub for Airflow 2.8

ingestion
1

Original Slack Thread

Hey, I have registered one request here as Datahub is using a very deprecated version of openlineage.
Can someone please check this? I guess no one has a user datahub-airflow v2 plugin with airflow 2.8. There are some major issues.
https://feature-requests.datahubproject.io/p/depricated-open-lineage-package-for-airflow-2-8

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: We’re experimenting with a new AI assistant for questions in the <#CUMUWQU66> channel to help you track down helpful resources more quickly. Keep in mind that responses may not always be spot-on; feel free to ask it follow-up questions and don’t forget to provide its responses a :thumbsup: or :thumbsdown: so other folks in the Community know if you’re still looking for help on the topic!

3

I’m not able to find any data sources to answer your question. Could you provide more details about the specific issue you are facing or the information you are seeking? If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.

4

What issues are you seeing? Could you provide a stack trace or other debug information?

5

Sure <@U01GZEETMEZ>.

ERROR - Failed to extract metadata You are trying to use `common-sql` with DatabricksHook, but its provider does not support it. Please upgrade the provider to a version that supports `common-sql`. The hook class should be a subclass of `airflow.providers.common.sql.hooks.sql.DbApiHook`. Got DatabricksHook Hook with class hierarchy: [&lt;class 'airflow.providers.databricks.hooks.databricks.DatabricksHook'&gt;, &lt;class 'airflow.providers.databricks.hooks.databricks_base.BaseDatabricksHook'&gt;, &lt;class 'airflow.hooks.base.BaseHook'&gt;, &lt;class 'airflow.utils.log.logging_mixin.LoggingMixin'&gt;, &lt;class 'object'&gt;]

6

package versions

acryl-datahub-airflow-plugin[plugin-v2]==0.13.3.3
apache-airflow-providers-common-sql==1.14.1```
7

What does your dag look like? I’m not sure this issue is specific to our airflow provider, as I see other reports of this issue e.g. https://stackoverflow.com/questions/77715057/how-to-resolve-common-sql-with-awsgenerichook-error

8

Seems like this indicates that the connection type is misconfigured

9

Hey, as you can see from the log, its correctly calling DatabricksHook and the connection type is also correct as databricks. In stack overflow they have wrongly configured i think.

10

What operator are you using this connection with?

11

Basically DatabricksHook does not inherit DbApiHook in any version, but common-sql requires it. And deprecated version of openlineage calls the common-sql.

12

We use multiple operators with this connection like DatabricksSqlOperator etc. But datahub plugin v2 directly takes care of it

13

I haven’t been able to repro this - could you provide the dag + connection setup you’re using?

14

Also I don’t believe the OL version that we use calls airflow-common-sql-provider. It internally has some methods in openlineage.common.sql, but those are just helper utils