Troubleshooting meta_mapping in dbt Cloud for DataHub Integration

Original Slack Thread

[Question about meta_mapping dbt-cloud]

Hello everyone,

I’m not sure if meta_mapping is working for dbt Cloud. I’ve tried several different configurations, but neither terms nor users were added to any model. However, within the table properties, I can correctly see the meta JSON I’ve defined in dbt (check picture).

Can anyone help me? Anyone having this issue?
Datahub Version: v0.13.3
I have the following dbt Cloud recipe:

    type: dbt-cloud
    config:
        account_id: ...
        project_id: ...
        job_id: ...
        target_platform: postgres
        stateful_ingestion:
            enabled: true
        token: ...
        target_platform_instance: postgres
        access_url: ...
        metadata_endpoint: ...
        node_name_pattern:
            deny:
                - '.*marts_.*'
            allow:
                - marts
        meta_mapping:
            datahub.data_governance.owner:
                match: '.*'
                operation: add_owner
                config:
                    owner_type: user
                    owner_category: BUSINESS_OWNER
            datahub.data_governance.team_owner:
                match: '.*'
                operation: add_term
                config:
                    term: '{{ $match }}'
            datahub.documentation_link:
                match: '.*'
                operation: add_doc_link
                config:
                    link: '{{ $match }}'
                    description: 'Data Model Link'```
Thanks![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F08ATEK72CE/image.png)

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

To address the issue with meta_mapping not working for dbt Cloud, let’s go through a few potential troubleshooting steps and configurations based on the provided data sources:

  1. Verify Meta Properties in dbt Models: Ensure that the meta properties you are trying to map (e.g., datahub.data_governance.owner, datahub.data_governance.team_owner, datahub.documentation_link) are correctly defined in your dbt models. The meta properties should be present in the dbt model’s meta section.

  2. Check Meta Mapping Configuration: According to the <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/dbt/dbt.md|DataHub documentation>, the meta_mapping configuration should match the keys in your dbt model’s meta properties. Ensure that the regex patterns in the match field are correctly capturing the intended values.

  3. Enable Meta Mapping: Make sure that the enable_meta_mapping option is set to true in your configuration. This ensures that the mappings defined in the meta_mapping section are applied during ingestion.

  4. Review Logs for Errors: Check the logs generated during the ingestion process for any errors or warnings related to meta_mapping. This can provide insights into what might be going wrong.

  5. Test with Simplified Configuration: Try simplifying your meta_mapping configuration to isolate the issue. For example, start with a single mapping and verify if it works, then gradually add more mappings.

  6. Check for Known Issues: Refer to the <https://github.com/datahub-project/datahub/issues|GitHub issues> for any known problems related to meta_mapping in dbt Cloud ingestion.

If these steps do not resolve the issue, please provide more details about the specific meta properties defined in your dbt models and any error messages you encounter during ingestion.

Would you like more detailed steps or further assistance on any specific part of this process?

Sources:

<@U07FV3UHE4D> help ?

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><