Migrating Data Between DataHub Instances: Sample Recipe and Steps

ingestion
1

Original Slack Thread

<@U06TM7M40RK>, I am testing out migrating data from one datahub instance to another. Can you give me the sample recipe?

2

To migrate data from one DataHub instance to another, you can use a recipe that defines both the source and sink configurations. The source will be your current DataHub instance, and the sink will be the target DataHub instance. Here is a sample recipe:

# Sample DataHub to DataHub migration recipe

# Source configuration: Current DataHub instance
source:
  type: datahub
  config:
    server: "http://<source_datahub_domain>:8080"
    token: "<source_api_token>"

# Sink configuration: Target DataHub instance
sink:
  type: datahub-rest
  config:
    server: "http://<target_datahub_domain>:8080"
    token: "<target_api_token>"

Steps to Use the Recipe:

  1. Replace Placeholders: Update <source_datahub_domain>, <source_api_token>, <target_datahub_domain>, and <target_api_token> with your actual DataHub instance details and API tokens.

  2. Run the Ingestion: Use the DataHub CLI to run the ingestion process with the recipe file. You can do this by executing the following command in your terminal:

    datahub ingest -c path/to/your/recipe.yml

This command will read metadata from the source DataHub instance and write it to the target DataHub instance.

For more information on configuring and running recipes, you can refer to the <https://datahubproject.io/docs/0.14.0/metadata-ingestion/cli-ingestion|DataHub CLI Ingestion documentation> and the <DataHub | DataHub source documentation>.

Would you like more detailed instructions or specific examples for any of these steps?

Sources:

3

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><