Understanding BigQuery Source Auditing with `use_exported_bigquery_audit_metadata`

ingestion
1

Original Slack Thread

<@U06TM7M40RK> for bigquery sources will use_exported_bigquery_audit_metadata extract usage metadata if I do not provide the service account with logging.logEntries.list or logging.privateLogEntries.list ?

2

To extract usage metadata using use_exported_bigquery_audit_metadata, you do not necessarily need the logging.logEntries.list or logging.privateLogEntries.list permissions. According to the <BigQuery | DataHub Ingestion Configuration>, when use_exported_bigquery_audit_metadata is set to true, lineage and usage information is computed using exported BigQuery logs. This means that the extraction process relies on the exported audit logs rather than directly accessing the logs via the logging API, which would require the mentioned permissions.

However, it is important to ensure that the service account has access to the datasets where the exported audit logs are stored. This typically involves permissions like bigquery.tables.getData and bigquery.jobs.create for accessing and querying the audit metadata datasets.

Would you like more detailed steps or additional information on setting up the service account for this purpose?

Sources:

  • <BigQuery | DataHub Ingestion Configuration> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
3

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><