Profiling a Specific Table in a Dataset with Data Ingestion Recipe

ingestion
1

Original Slack Thread

Hi team, quick question, if I wish to ingest a dataset and profile a specific table (that is not in the dataset I am ingesting). Can I use a single recipe or will I need 2 recipes?
Would this format work?

  type: bigquery
  config: 
    project_id_pattern:
      allow:
        - .*project1
    schema_pattern:
      allow:
      - dataset1.*
    
    include_usage_statistics: false

    profiling:
      enabled: true
      profile_table_size_limit: null
      profile_table_row_limit: null
    profile_pattern:
      allow:
        - dataset2.table1.*

    stateful_ingestion:
        enabled: true
        state_provider:
          type: "datahub"
          config:
            datahub_api:
              server: "<http://gott-datahub-gms:8080>"
pipeline_name: "bigquery-datahub-submit"
sink:
  type: "datahub-rest"
  config:
    server: "<http://gott-datahub-gms:8080>"```
2

<@UV14447EU> might help you
cc: <@U04N9PYJBEW>

3

We won’t profile a table if it’s not ingested normally (i.e. if it doesn’t fit the schema_pattern and other patterns). I would just ingest your dataset and the specific table you want. You should be able to do this via table_pattern, with something like this:

  allow:
    - .*dataset1.*
    - .*dataset2.*
table_pattern:
  allow:
    - .*dataset1\..*
    - .*dataset2\.table1.*
profile_pattern:
  ...```