Installing Plugins for Data Sources in DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> I ran through the quickstart guide and trying to add a new source, but half the data sources shown in the docs don’t show up. How do I install them?

2

To add new data sources that are not showing up in the UI, you may need to install the necessary plugins for those sources. Here’s how you can do it:

  1. Identify the Required Plugin: Each data source in DataHub may require a specific plugin. You can find the list of available plugins and their installation commands in the <https://datahubproject.io/docs/0.13.1/cli#sources|DataHub CLI documentation>.

  2. Install the Plugin: Use the pip command to install the required plugin. For example, if you want to add support for BigQuery, you would run:

    pip install 'acryl-datahub[bigquery]'

  3. Verify Plugin Installation: After installing the plugin, you can verify that it is installed by running:

    datahub check plugins

  4. Restart DataHub: If you are running DataHub using Docker, you may need to restart your DataHub instance to ensure the new plugins are recognized.

  5. Use CLI for Ingestion: If the UI does not support certain sources, you can use the CLI to ingest metadata. Create a YAML recipe file specifying the source and sink configurations, and run the ingestion using:

    datahub ingest -c &lt;path_to_recipe_file.yml&gt;

If you need more detailed steps or assistance with a specific part of the process, feel free to ask!

Sources:

3

Hi <@U08AR1CJX1U>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

4

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><