Troubleshooting Snowflake metadata ingestion with schema exclusion filters

ingestion
1

Original Slack Thread

Hi DataHub Team,
I am trying to ingest the Snowflake metadata information from a particular database but I want to refrain from ingesting any schema that ends in *_DEV or *_DEMO or *_QA. I have created the ingestion recepie file as provided below. While execution the ingestion is failing and it says could not find any schemas or tables. Can you please point me out where am I getting it wrong

    type: snowflake
    config:
        account_id: Snowflake_Account_redacted
        include_table_lineage: true
        include_view_lineage: true
        include_tables: true
        include_views: true
        profiling:
            enabled: true
            profile_table_level_only: false
        stateful_ingestion:
            enabled: false
        warehouse: Warehouse_redacted
        username: User_redacted
        password: '${Snowflake_pwd}'
        role: Role_Redacted
        database_pattern:
            allow:
                - DB_I_WANT_TO_INGEST
        schema_pattern:
            deny:
                - '*_DEV'
                - '*_TEST'
                - '*_DEMO'
sink:
    type: datahub-rest
    config:
        server: '<http://datahub-datahub-gms.datahub.svc.cluster.local:8080>'```
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

• UI
• 0.13.0
• Snowflake