How to Retrieve a List of URNs from a Specific Data Source Using the Terminal

ingestion
1

Original Slack Thread

<@U06TM7M40RK> How can I get a list of all urns from a specific data source, from terminal?

2

To get a list of all URNs from a specific data source using the terminal, you can use the DataHub GraphQL API to perform a search query filtered by the data source platform. Here’s a step-by-step guide on how to do this:

  1. GraphQL Query: You can use a GraphQL query to search for datasets by specifying the platform. For example, if you want to list all dataset URNs from a platform like “redshift”, you can use the following query:

    query getUrnsByPlatform {
  search(input: {type: DATASET, query: "*", start: 0, count: 100, orFilters: [{and: [{field: "platform", values: ["redshift"], condition: CONTAIN}]}]}) {
    start
    count
    total
    searchResults {
      entity {
        urn
      }
    }
  }
}

  2. Execute the Query: You can execute this query using a curl command in the terminal. Make sure to replace YOUR_DATAHUB_TOKEN with your actual DataHub token and YOUR_GRAPHQL_ENDPOINT with your DataHub GraphQL endpoint URL.

    curl --location --request POST 'YOUR_GRAPHQL_ENDPOINT' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_DATAHUB_TOKEN' \
--data-raw '{
  "query": "query getUrnsByPlatform { search(input: {type: DATASET, query: \"*\", start: 0, count: 100, orFilters: [{and: [{field: \"platform\", values: [\"redshift\"], condition: CONTAIN}]}]}) { start count total searchResults { entity { urn } } } }",
  "variables": {}
}'

This will return a list of URNs for datasets from the specified platform.

Would you like more detailed steps or assistance with a specific part of this process?

Sources:

  • <Slack Message with GraphQL Query for Platform> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
3

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><