Using OAuth Tokens for BigQuery Authentication with AWS Workload Identity

Original Slack Thread

Hello Team, I see for the bigquery auth the ingestion requires key json file. Is there a way to do using the token. We are calling bigquery from AWS side and using workload identity to access the bigquery.

<@U04NA0ZKQDD>
You can add detail in recipe , see this example https://datahubproject.io/docs/generated/ingestion/sources/bigquery#create-a-service-account-in-the-extractor-project

We have CLOUDSDK_AUTH_ACCESS_TOKEN which is not a part of default chain of possible authentication on gcp. We make the client something like this

client = bigquery.Client(
    credentials=credentials,
    project=PROJECT_NAME
)```

<@U04NA0ZKQDD>
As per doc looks like this flow is not supported. <@U04N9PYJBEW> might help you

I think this could be a new feature and gcp recommends temporary tokens when working cross cloud as sevice_account key are prone security vulnerabilities

You can go for feature request on GitHhub

<@UV14447EU> please take a look here if you get a chance. I get the feeling we can support this with either extra_client_options but am not sure

Currently we don’t path in any credential object to the client but we either set (if you specify credentials in the config) or rely on the GOOGLE_APPLICATION_CREDENTIALS environment variable.
If oauth based auth can’t be set in an application credential file then our bigquery source needs to be extended to support.

For my case I changed somepart of bigquery code and able to conenct to bigquery. https://github.com/rjtshrm/datahub/pull/1/files. If you think this is something can be added to datahub code then I can extend raising PR there with more details.