Troubleshooting DBT Meta Mapping Functionality Issues

ingestion
1

Original Slack Thread

Hello,
I’m experiencing issues with DBT ingestion. It seems the meta_mapping functionality isn’t working beyond the default configuration. When I ran the ingestion for the first time, it successfully tagged all my tables with the tags I had in DBT. However, I’m unable to perform any other actions. Whenever I try to add terms, assign users, or make similar updates, nothing happens.

My DataHub version: v0.13.2

    type: dbt-cloud
    config:
        account_id: 'account_id'
        project_id: 'project_id'
        job_id: 'job_id'
        target_platform: postgres
        stateful_ingestion:
            enabled: true
        token: '${DBT_METADATA_RO_TOKEN}'
        target_platform_instance: postgres
        access_url: '<https://abcd123.xxx.dbt.com>'
        metadata_endpoint: '<https://abcd123.metadata.xxx.dbt.com/graphql>'
        node_name_pattern:
            deny:
                - '.*marts_.*'
        meta_mapping:
            datahub.data_governance.owner:
                match: '.*'
                operation: add_owner
                config:
                    owner_type: user
                    owner_category: BUSINESS_OWNER
            datahub.data_governance.team_owner:
                match: '.*FinanceTeam.*'
                operation: add_term
                config:
                    term: FinanceTeam
            datahub.documentation_link:
                match: '.*'
                operation: add_doc_link
                config:
                    link: '{{ $match }}'
                    description: 'Data Model Link'```
Thanks !
2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

To address the issues you’re experiencing with the meta_mapping functionality in your DBT ingestion setup, let’s break down the potential solutions and checks you can perform:

  1. Verify Meta Mapping Configuration: Ensure that your meta_mapping configuration is correctly set up. According to the <https://datahubproject.io/docs/0.14.0/generated/ingestion/sources/dbt#dbt-meta-automated-mappings|DataHub documentation on dbt meta automated mappings>, the meta_mapping section should be defined properly with the correct operations and configurations. Double-check the syntax and ensure that the operations (add_owner, add_term, add_doc_link) are correctly specified.

  2. Check for Regex Matching: The match field in your meta_mapping uses regex to determine which meta properties to apply the operations to. Ensure that the regex patterns are correctly matching the intended meta properties in your DBT models. You can test these regex patterns separately to confirm they match the expected values.

  3. Review Metadata Endpoint and Access URL: Ensure that the access_url and metadata_endpoint are correctly configured. According to the <https://datahubproject.io/docs/0.14.0/generated/ingestion/sources/dbt|DataHub documentation>, these URLs are crucial for accessing the DBT Cloud APIs and fetching metadata. Verify that these URLs are correct and accessible.

  4. Enable Debug Logging: Run the DataHub CLI with debug logs enabled to get more insights into what might be going wrong. Use the --debug flag to enable detailed logging:

    datahub ingest -c your_ingestion_recipe.yml --debug

  5. Check for Known Issues and Updates: There might be known issues with the version of DataHub or the dbt ingestion plugin you are using. Consider checking for any updates or patches that might address these issues.

  6. Inspect DBT Artifacts: Ensure that the DBT artifacts (manifest.json, catalog.json, sources.json, run_results.json) are correctly generated and contain the expected metadata. Manually inspect these files to ensure they include the necessary information about your models.

Would you like more detailed steps or further assistance on any specific part of this process?

Sources:

4

Hi, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

5

Is there maybe any config issues in my yaml recipe ? <@U07FV3UHE4D>

6

My dbt config:

models:
  data_warehouse:
      marts:
        materialized: table
        +tags: marts
        +persist_docs:
          relation: true
          columns: true
        finance:
          +meta:
            datahub:
              data_governance:
                team_owner: "FinanceTeam"```
7

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><

8

<@U07FV3UHE4D> help !