Adding Domains to Containers in DataHub using the `pattern_add_dataset_domain` Transformer

[user-3]

Original Slack Thread

<@U06TM7M40RK> I am ingesting databases using the standard glue ingestion. I add domains using the add_dataset_domain transformer. However the domain is not added to the databases which are created in datahub by the glue ingestion. I can see in the codebase that there is a variable container_domain_mapping at the end of AddDatasetDomain but it doesn’t seem to actually work. Do you have advice for how to apply domains to containers?

[user-1]

To apply domains to containers in DataHub, you can use the pattern_add_dataset_domain transformer with the is_container field set to true. This configuration will ensure that domains are attached not only to datasets but also to their associated containers. Here’s how you can configure it:

1. Configure the Transformer: Use the pattern_add_dataset_domain transformer in your ingestion recipe and set the is_container field to true.

2. Define the Domain Pattern: Specify the regex pattern that matches the URNs of the datasets and containers you want to assign domains to.

Here is an example configuration:

transformers:
  - type: "pattern_add_dataset_domain"
    config:
      is_container: true
      semantics: OVERWRITE
      domain_pattern:
        rules:
          'urn:li:dataset:\(urn:li:dataPlatform:glue,.*': ["urn:li:domain:your_domain"]

This configuration will ensure that both datasets and their containers created by the Glue ingestion are assigned to the specified domain in DataHub.

If you are using a different method or need more specific instructions, please let me know!

Would you like more detailed instructions or have any specific questions about this process?

Sources:

• <Dataset | DataHub Add Dataset Domain Transformer Documentation>
• <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/transformer/dataset_transformer.md|GitHub Dataset Transformer Documentation> 1 button 0 button Hint: Mention <@U06TM7M40RK> in the thread for followups.

[user-3]

What about adding tags to containers?

[datahub_team]

Hi ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look

Hope this helps ! ><