Disabling Redshift Automatic Lineage in DataHub Configuration

getting-started
1

Original Slack Thread

Hello Everyone,
Just wanted to check is there any way to disable automated lineages created in Redshift in DataHub.
Thanks!

2

Yes, table lineage, view lineage and column-level lineage are all configurable within the provided redshift integration:
https://datahubproject.io/docs/generated/ingestion/sources/redshift/#config-details

3

Hello <@U05JJ9WESHL>, Thanks for the reply.
Could you please highlight which specific option will help me in disabling Redshift automatic lineage created with views and how can we disable it.

I have tried to exclude “include_table_lineage” option but its still creating the automatic lineage. Also have tried all the below 3 lineage collector types :-
stl_scan_based
sql_based
mixed
But its still not working.

Also I have tried to exclude “include_view_lineage” option in YAML recipe as below but its still not working. I am getting the error as "Failed to configure Redshift.
Extra fields not permitted “include view lineage”.

Kindly suggest if there is any specific thing which is required to be performed.attachment

4

Extra fields not permitted "include view lineage". - I usually get this error when I accidentally add a field outside of the config section.

After clearing your current redshift data with the CLI, Try the following:

    type: redshift
    config:
        include_table_lineage: false
        include_table_location_lineage: false
        extract_column_level_lineage: false
        include_unload_lineage: false
        include_view_lineage: false
        include_view_column_lineage: false
        include_copy_lineage: false
        {...}```
5

Hello <@U05JJ9WESHL>,

Thanks for the help, I have tried to delete the complete Redshift data and included the above recipe in the config section and then run the ingestion still I am getting the below error.

extract_column_level_lineage
  extra fields not permitted (type=value_error.extra)
include_view_column_lineage
  extra fields not permitted (type=value_error.extra)
include_view_lineage
  extra fields not permitted (type=value_error.extra)```
![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F069YP3JBS6/image.png?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
6

hmm, the docs may need an update then. With those removed, does the view lineage still render?
To me, the code reads like having include_table_lineage and include_copy_lineage set to false will skip all lineage, even views:
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py#L420C61-L420C81

7

So I would also set include_copy_lineage to false :slightly_smiling_face: . Missed that one.

8

Thank you so much <@U05JJ9WESHL>! This solution worked for me.

9

Nice! Glad we were able to figure this one out :+1: