Cleaning Up Existing Spark Pipelines and Tasks in DataHub Instance

ingestion
1

Original Slack Thread

<@U06TM7M40RK> how to cleanup all the existing pipelines and tasks from Spark that I have in my datahub instance? I have disabled the datahub-spark integration but want to clean up the existing metadata as well.

2

To clean up all existing pipelines and tasks from Spark in your DataHub instance, you can use the DataHub CLI to delete the relevant metadata. Here are the steps you can follow:

  1. Use the DataHub CLI: You can delete metadata attached to entities by providing specific URNs or using filters to identify a set of URNs. This can be done using the datahub delete command.

  2. Soft Delete: By default, the delete command performs a soft delete, which hides the entity from the UI but retains the metadata. You can perform a soft delete using a specific URN or a filter. For example:

    datahub delete --platform spark --soft

  3. Hard Delete: If you want to permanently remove the metadata, you can perform a hard delete. This action is irreversible, so ensure you want to delete all data associated with the entity. For example:

    datahub delete --platform spark --hard

  4. Recursive Deletes: If you have container or dataPlatformInstance entities, you can perform recursive deletes:

    datahub delete --urn "urn:li:container:<your-container-urn>" --recursive

  5. Dry Run: Always use the --dry-run option to test your delete command before executing it to ensure you are deleting the correct entities.

  6. Force Delete: Use the --force option to skip confirmation prompts if you are sure about the deletion.

For more detailed instructions, you can refer to the <https://datahubproject.io/docs/0.14.0/how/delete-metadata|DataHub documentation on deleting metadata>.

Would you like more detailed guidance on any of these steps?

Sources:

3

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><