Ingesting MLflow Data from Databricks into DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> ingest mlflow from databricks

2

To ingest MLflow data from Databricks into DataHub, you can use the MLflow ingestion source provided by DataHub. Here are the key steps and configurations you need to follow:

  1. Install the MLflow Plugin: Ensure that you have the acryl-datahub package installed with MLflow support. You can install it using the following command:

    pip install 'acryl-datahub[mlflow]'

  2. Create an Ingestion Recipe: You need to create a YAML file that defines the source and sink for metadata ingestion. Here is an example configuration for MLflow:

    source:
  type: mlflow
  config:
    tracking_uri: "<your_tracking_uri>"
    registry_uri: "<your_registry_uri>"
    model_name_separator: "_"
    env: "PROD"

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"
    • tracking_uri: This is the URI of your MLflow tracking server.
    • registry_uri: This is the URI of your MLflow model registry server. If not set, it defaults to the tracking_uri.
    • model_name_separator: A string that separates the model name from its version.
    • env: The environment that all assets produced by this connector belong to.

  3. Run the Ingestion: Use the DataHub CLI to run the ingestion based on your recipe file.

    datahub ingest -c path/to/your/recipe.yml

  4. Authentication: If your MLflow server requires authentication, you may need to set environment variables such as MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD for basic authentication, as suggested in <https://github.com/datahub-project/datahub/issues/12850|this GitHub issue>.

For more detailed information on MLflow integration, you can refer to the <https://datahubproject.io/docs/0.14.0/generated/ingestion/sources/mlflow|MLflow documentation>.

Would you like more detailed steps or additional information on any specific part of the process?

Sources:

3

Hi <@U08AER1JC7Q>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

4

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><