Troubleshooting Prefect-DataHub Integration Issues

Original Slack Thread

Hi y’all, we use Prefect and I’m looking at using DataHub. I would love to use <https://datahubproject.io/docs/lineage/prefect/|this integration>, but it doesn’t look like it actually exists? In particular:
• I can’t run pip install 'prefect-datahub'
• It’s <https://pypi.org/project/prefect-datahub/|not on PyPI>
• <https://github.com/acryldata/prefect-datahub|The repository seems empty on Github>, but has a commit that seems to implement everything?

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

It seems likely that the integration you’re referring to is indeed mentioned in the <https://github.com/datahub-project/datahub/blob/master/docs/lineage/prefect.md|DataHub documentation>, which outlines how to use Prefect with DataHub. However, if you’re unable to find the prefect-datahub package on PyPI, it might be due to a discrepancy between the documentation and the actual availability of the package.

Based on the <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion-modules/prefect-plugin/README.md|Prefect Plugin README>, the prefect-datahub package is intended to emit flows and tasks metadata to the DataHub REST API. The documentation suggests installing it via pip, but if it’s not available on PyPI, there might be an issue with the package’s publication or naming.

To proceed, you might want to:

  1. Verify if the package is available under a different name or if there are any updates in the DataHub repository regarding its availability.
  2. Check the <https://github.com/datahub-project/datahub|DataHub GitHub repository> for any issues or discussions related to the prefect-datahub package.
  3. Consider reaching out to the DataHub community or maintainers for clarification on the package’s availability.

Would you like more detailed guidance on any of these steps?

Sources:

Hi, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

<@U07FV3UHE4D> plz help :slight_smile: