Running SQL Queries with DataHub v12 using CLI-based Ingestion

ingestion
1

Original Slack Thread

Hello <@U01GZEETMEZ> <@U02G4B6ADL6> <@U02H9D09GSH> I am using v12 of datahub and have sql queries in a json format based on this link https://datahubproject.io/docs/generated/ingestion/sources/sql-queries/. How am I suppose to actually run these queries? Can someone guide or give some documentation on this.

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

It is used with CLI-based ingest

  1. create Query file queries.json with queries & urn as shown in docs
  2. Install plugin pip install 'acryl-datahub[sql-queries]'
  3. create recipe file ingest_queries.yml like below
  server: <http://localhost:8080>
  timeout_sec: 60
source:
  type: sql-queries
  config:
    platform: "snowflake"
    default_db: "SNOWFLAKE"
    query_file: "./queries.json"```
4. run cli ingest
```datahub ingest -c ingest_queries.yml```
<https://datahubproject.io/docs/generated/ingestion/sources/sql-queries/#cli-based-ingestion>
<https://datahubproject.io/docs/cli/>