Preventing Overwriting of Detailed Descriptions in Ingestion Pipeline

[user-1]

Original Slack Thread

<@U06TM7M40RK> There is a dataset which is updated by two pipeline. The one is for all bigquery tables and the other is for Tier 1 tables.
The former will ingest a simple description to the documentation area of the table, and the latter will ingest a detailed description to the documentation area of the table.
I’m worried about the former ingestion could overwrite a detailed description with a simple description.

[user-3]

To prevent the former ingestion pipeline from overwriting the detailed descriptions with simple descriptions, you can use the PATCH semantics in your ingestion configuration. This approach allows you to update only specific fields without overwriting existing metadata.

Here’s how you can configure your ingestion recipe to use PATCH semantics:

1. Use PATCH Semantics: Ensure that the ingestion recipe for the pipeline that ingests simple descriptions is configured to use PATCH semantics. This will prevent it from overwriting existing detailed descriptions.

2. Configure the Ingestion Recipe: Modify the ingestion recipe to include the PATCH semantics for the description field.

Here’s an example of how you can configure your ingestion recipe:

source:
  type: bigquery
  config:
    # Your BigQuery source configuration
transformers:
  - type: simple_add_dataset_properties
    config:
      properties:
        description: "Simple description"
      semantics: PATCH
sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"

In this configuration:

• The semantics: PATCH setting ensures that the simple description is added only if there is no existing description, thus preserving any detailed descriptions already present.

Would you like more detailed steps or further assistance on any specific part of this process?

Sources:

• https://datahubspace.slack.com/archives/CUMUWQU66/p1726216517.161159|https://datahubspace.slack.com/archives/CUMUWQU66/p1726216517.161159
• https://datahubproject.io/docs/0.13.1/generated/metamodel/entities/dataset#editabledatasetproperties|https://datahubproject.io/docs/0.13.1/generated/metamodel/entities/dataset#editabledatasetproperties 0 button 0 button Hint: Mention @runllm in the thread for followups.

[datahub_team]

Hi ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look

Hope this helps ! ><