Discussion on Looker API Change and DataHub ingestion methods

ingestion
1

Original Slack Thread

Hi everyone: There is a mandatory Looker API Change coming down.
https://cloud.google.com/looker/docs/best-practices/query-id-update-instructions
Anyone know if the DH Ingestion is using <query_id> or GET /queries/slug<slug> ?

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

I actually see the following in the looker_source.py module:

all_looks: List[Look] = self.looker_api.all_looks(
fields=look_fields, soft_deleted=self.source_config.include_deleted
)
for look in all_looks:
if look.id in self.reachable_look_registry:
# This look is reachable from Dashboard
continue

        if look.query_id is None:
            <http://logger.info|logger.info>(f”query_id is None for look {look.title}({look.id})“)
            continue

        query: Query = self.looker_api.get_query(look.query_id, query_fields)
4

But, I do see this on line 1164-1188 of looker_source.py Getting slug as the last query field.

def extract_independent_looks(self) -> Iterable[MetadataWorkUnit]:
“”"
Emit MetadataWorkUnit for looks which are not part of any Dashboard
“”"
if self.source_config.extract_independent_looks is False:
return

    self.reporter.report_stage_start(“extract_independent_looks”)

    logger.debug(“Extracting looks not part of Dashboard”)
    look_fields: List[str] = [
        “id”,
        “title”,
        “description”,
        “query_id”,
    ]
    query_fields: List[str] = [
        “id”,
        “view”,
        “model”,
        “dynamic_fields”,
        “filters”,
        “fields”,
        “slug”,
    ]
5

From what I can tell, we don’t ever directly query queries (or render_tasks or query running queries) so I think we will be okay here. Thanks for the heads up!

6

Thanks Andrew, I am looking into this further just to confirm.