Hi , I have made a connection with my airflow to datahub. I am working on data lineage part. Not sure but whenever datahub goes down, the airflow also goes down . Does anyone know about this issue?
Hey there! Make sure your message includes the following information if relevant, so we can help more effectively!
- Which DataHub version are you using? (e.g. 0.12.0)
- Please post any relevant error logs on the thread!
0.12.0
It’s strange that airflow is going down, it’s highly unlikely!. It is possible that an airflow task might fail due to an inability to emit metadata to datahub.
Take a close look at airflow service logs and task logs which are running state at the time of failure
which airflow integration you are using plugin v1 or plugin v2 ?
if you are using plugin v1 enable graceful_exceptions = true
to suppress lineage backend error and will not cause the overall task to fail.