Ensuring Proper Configuration for Column-Level Lineage in DataHub

Original Slack Thread

include_table_lineage: true
how can in include more colume_level?

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

To include column-level lineage in DataHub, you need to ensure that your ingestion source supports this feature and that the necessary configurations are enabled. Based on the <https://datahubproject.io/docs/0.13.1/generated/ingestion/sources/dbt|DataHub documentation for dbt>, here are the steps you can follow:

  1. Enable Column-Level Lineage: Ensure that the include_column_lineage configuration is set to true in your ingestion source configuration. This setting is often enabled by default but requires other settings to be correctly configured.

  2. Check Additional Configurations: For dbt, you also need to enable infer_dbt_schemas. For other sources, ensure that any required settings for column-level lineage are enabled.

  3. Verify Artifacts: Make sure that all necessary artifacts (e.g., manifest file, catalog file) are correctly provided and accessible.

  4. Update DataHub Version: If you are using an older version of DataHub and not seeing column-level lineage, consider updating to a newer version, as there might be bug fixes or improvements related to this feature.

Here is an example configuration for dbt:

source:
  type: "dbt"
  config:
    include_column_lineage: true
    infer_dbt_schemas: true
    # other configurations

If you are using a different data source, please refer to the specific <https://datahubproject.io/integrations|source documentation> for that source to ensure all necessary configurations are enabled.

Would you like more detailed steps or troubleshooting tips for a specific data source?

Sources:

Hi, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><