Using datahub cli to delete projects and datasets ending in 'data-test'

troubleshoot
1

Original Slack Thread

Any idea how I can use datahub cli to delete projects and corresponding datasets within those projects?
I have project names like:
-data-test
-data-prod
abc-data-xyz
test-abc-xyz
I need to delete only the ones ending in ‘data-test’. Someone ingested test data into prod long time ago and we need to clean it up.

I’m using dry-run to verify what would be deleted and no matter what combination I try putting in the query I end up with more than I’m looking for.
This is skeleton I’m working with:
datahub delete --env PROD --platform bigquery --entity-type dataset --query <what can I put here> --dry-run
I expected that --query data\-test would work but it does not.

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

Hi Jasminko! <@U01JUBAPZPS> might be able to help you with this one

4

Hey <@U064A61UV8R> definitely can help here. Did you try --query "data-test" ? so using quotes

5

Hi <@U01JUBAPZPS> I sure did. But it returns way more results than I expect it to. For example I get abc-def-xyz (no data and no test in the name). I need a way to identify instances where the first part of the name ends with ‘data-test’. This part corresponds to our BigQuery project.
urn:li:dataset:(urn:li:dataPlatform:bigquery,<something>-data-test.<something>.<something else>,PROD)