Enabling Metadata Service Authentication in DataHub GMS Container

ingestion
1

Original Slack Thread

hello, I am trying to enable metadat ingestion service auth within GMS, but the documentations is not up to date I believe, and i cannot find application.yml within datahub-gms comtainer, nor i could find the METADAT_SERVICE_AUTH_ENABLED environment variable within the containers

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: _Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

To enable metadata ingestion service authentication within the DataHub GMS, you need to set the METADATA_SERVICE_AUTH_ENABLED environment variable to true for both the datahub-gms and datahub-frontend containers. If you cannot find the application.yml file or the environment variable within the containers, you can follow these steps:

Steps to Enable Metadata Service Authentication

Docker Compose Deployment

  1. Edit docker-compose.yml File:

    • Locate your docker-compose.yml file and add the METADATA_SERVICE_AUTH_ENABLED environment variable to the datahub-gms and datahub-frontend services.
    services:
  datahub-gms:
    environment:
      METADATA_SERVICE_AUTH_ENABLED: "true"
  datahub-frontend:
    environment:
      METADATA_SERVICE_AUTH_ENABLED: "true"

  2. Restart Docker Services:

    • Restart the services to apply the changes.
    docker-compose down
docker-compose up -d

Kubernetes Deployment

  1. Edit Helm Values:

    • If you are using Helm for Kubernetes deployment, set global.datahub.metadata_service_authentication.enabled to true in your values.yaml file.
    global:
  datahub:
    metadata_service_authentication:
      enabled: true

  2. Upgrade Helm Release:

    • Apply the changes by upgrading your Helm release.
    helm upgrade <your-release-name> datahub/datahub -f values.yaml

Example Configuration

Here is an example of how your docker-compose.yml might look:

version: '3.5'
services:
  datahub-gms:
    image: linkedin/datahub-gms
    environment:
      METADATA_SERVICE_AUTH_ENABLED: "true"
    ports:
      - "8080:8080"

  datahub-frontend:
    image: linkedin/datahub-frontend
    environment:
      METADATA_SERVICE_AUTH_ENABLED: "true"
    ports:
      - "9002:9002"

Additional Configuration

According to the <https://github.com/datahub-project/datahub/blob/master/docs/authentication/introducing-metadata-service-authentication.md|DataHub documentation>, enabling metadata service authentication requires additional steps for handling ingestion and API requests.

  1. Generate Personal Access Token:

    • After enabling authentication, generate a Personal Access Token (PAT) via the DataHub UI.
    • Navigate to Settings > Access Tokens > Generate Personal Access Token.
    • Save the generated token securely.

  2. Update Ingestion Recipes:

    • Update your ingestion recipes to include the generated token.
    source:
  # source configs
sink:
  type: "datahub-rest"
  config:
    server: 'http://datahub-gms:8080'
    token: <your-personal-access-token-here!>

  3. Use the Token in API Requests:

    • Include the token in the Authorization header for API requests.
    curl 'http://localhost:9002/api/gms/entities/urn:li:corpuser:datahub' -H 'Authorization: Bearer <access-token>'
4

Would you like more detailed steps or assistance with a specific part of this process?

Sources:

5

I have tried doing this, but it is not working still