How to Include More Column-Level Lineage in Ingestion Config for ClickHouse

ingestion
1

Original Slack Thread

include_table_lineage: true
how can in include more colume_level?
in ingestion config for clickhouse?

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

To include column-level lineage in your ingestion configuration for ClickHouse, you need to ensure that your configuration is set up to capture this level of detail. Based on the <https://datahubproject.io/docs/0.14.0/generated/ingestion/sources/clickhouse|ClickHouse ingestion documentation>, the ClickHouse source supports extracting metadata for tables, views, materialized views, and dictionaries, as well as table, row, and column statistics via optional SQL profiling.

However, the documentation does not explicitly mention a parameter like include_column_lineage for ClickHouse, as it does for other sources like dbt. This suggests that column-level lineage might not be directly supported or might require additional configuration or custom development.

Here are some general steps you can take to attempt to include column-level lineage:

  1. Enable SQL Profiling: Ensure that SQL profiling is enabled in your ClickHouse ingestion configuration. This might help in capturing more detailed metadata, which could include column-level details.

  2. Check for SQL Parser Support: If your setup involves SQL queries, you might need to use DataHub’s SQL parser to extract column-level lineage. The <https://github.com/datahub-project/datahub/blob/master/docs/lineage/sql_parsing.md|SQL Parsing documentation> provides more details on how to use the SQL parser for this purpose.

  3. Custom Development: If column-level lineage is critical and not supported out of the box, you might need to develop a custom solution or contribute to the DataHub project to add this feature.

Would you like more detailed steps or troubleshooting tips on any of these points?

Sources:

4

Hi, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><