Automating BigQuery Lineage and Metadata Ingestion in Data Cataloguing

Original Slack Thread

Hi teams,
I am quite new in data cataloguing. I managed, following the quickstart webpage, to ingest my BigQuery dataset through the UI and I wanted to know if there was a way to automatise the ingestion of lineage (tables, columns) and metadata? Also the queries don’t appear, do they have to be saved in project or public?
Thanks for your help :slightly_smiling_face:

Hi - you could configure lineage extraction in recipe, and from 0.10.5 we support column-level lineage for BigQuery! : https://datahubproject.io/docs/generated/ingestion/sources/bigquery

Also, Queries/Stats tab is greyed out if there is no usage statistics for that dataset or there were no ingestion with usage extraction run before.