Issues with Datahub Release v0.13.0 in Redshift Connector with `use_lineage_v2` Flag

troubleshoot
1

Original Slack Thread

Hi Guys -

We have seen an issue with datahub release v0.13.0 in the redshift connector while using the use_lineage_v2 enabled flag. The use case we were trying to solve was to show the upstream lineage when there is temporary table involved. I believe this issue was fixed with this latest release. https://github.com/datahub-project/datahub/issues/4895
https://github.com/datahub-project/datahub/pull/9704
We were hoping to see the table A as upstream for table C when A–>temp B–>C. But this is showing some weird results.
We see 2 issues:

  1. For my main table f_order_event, the previous upstream lineage count showed 7 tables and they also showed in the lineage graph. However we got to know that there are some temporary tables also involved into creating f_order_event. We enabled the use_lineage_v2 flag on release v13 and now the count says 9 upstream but those extra 2 tables dont show in the lineage graph. Which is very confusing. Those extra tables are also not present in the upstreamLineage aspect in metadata storage. I confirmed that all permanent tables involved in the lineage for f_order_event are present in our datahub instance.
  2. THe 2nd issue is when we expand the up/down lineage to 2nd or 3rd levels we see the spinner loading icon permanently like its trying to fetch the information, but the 2nd/3rd level lineage already comes up on the screen. THis is not an issue with other connectors with large lineage like snowflake or vertica.
    <@U01GZEETMEZ> <@UV14447EU> any suggestions here? Our customers want to see the upstream tables responsible for creating the temp tables in the lineage, fine with not able to see temp tables.attachment
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

![attachment]({‘ID’: ‘F06TVA0S25R’, ‘EDITABLE’: False, ‘IS_EXTERNAL’: False, ‘USER_ID’: ‘U04UNS1T1JT’, ‘CREATED’: ‘2024-04-11 13:09:53+00:00’, ‘PERMALINK’: ‘Slack’, ‘EXTERNAL_TYPE’: ‘’, ‘TIMESTAMPS’: ‘2024-04-11 13:09:53+00:00’, ‘MODE’: ‘hosted’, ‘DISPLAY_AS_BOT’: False, ‘PRETTY_TYPE’: ‘PNG’, ‘NAME’: ‘image.png’, ‘IS_PUBLIC’: True, ‘PREVIEW_HIGHLIGHT’: None, ‘MIMETYPE’: ‘image/png’, ‘PERMALINK_PUBLIC’: ‘https://slack-files.com/TUMKD5EGJ-F06TVA0S25R-f54aed9161’, ‘FILETYPE’: ‘png’, ‘EDIT_LINK’: None, ‘URL_PRIVATE’: ‘Slack’, ‘HAS_RICH_PREVIEW’: False, ‘TITLE’: ‘image.png’, ‘IS_STARRED’: False, ‘PREVIEW_IS_TRUNCATED’: None, ‘URL_PRIVATE_DOWNLOAD’: ‘Slack’, ‘PREVIEW’: None, ‘PUBLIC_URL_SHARED’: False, ‘MESSAGE_TS’: ‘1712840996.725889’, ‘PARENT_MESSAGE_TS’: ‘1712840728.390889’, ‘MESSAGE_CHANNEL_ID’: ‘C029A3M079U’, ‘_FIVETRAN_DELETED’: False, ‘LINES_MORE’: None, ‘LINES’: None, ‘SIZE’: 106812, ‘_FIVETRAN_SYNCED’: ‘2024-04-14 12:55:22.653000+00:00’})