Configuring DataHub Sink Recipe to Point to GMS Ingress

troubleshoot
1

Original Slack Thread

<@U06TM7M40RK> how to configure sink recipe to point to gms ingress?

2

To configure a sink recipe to point to the GMS (Graph Metadata Service) ingress in DataHub, you need to set up the sink configuration in your recipe YAML file to use the datahub-rest sink type. This involves specifying the server URL where your DataHub GMS is accessible. Here’s a general outline based on the available data sources:

  1. Determine the GMS Endpoint: The server URL should point to the GMS endpoint. If you are using a hosted DataHub Cloud instance, the URL will be in the format https://<your-instance>.acryl.io/gms. If you are running DataHub locally or in a containerized environment, the URL might be something like http://localhost:8080 or http://datahub-gms:8080 depending on your setup.

  2. Configure the Sink: In your recipe YAML file, configure the sink section to use the datahub-rest type and specify the server URL. If authentication is required, include a token.

Here is an example configuration:

sink:
  type: "datahub-rest"
  config:
    server: "https://<your-instance>.acryl.io/gms"  # Replace with your GMS endpoint
    token: "<your_personal_access_token>"  # Optional, if authentication is required
  1. Additional Configuration Options: You can also configure additional options such as timeout_sec, retry_max_times, and disable_ssl_verification if needed. For example:
sink:
  type: "datahub-rest"
  config:
    server: "https://<your-instance>.acryl.io/gms"
    token: "<your_personal_access_token>"
    timeout_sec: 60
    retry_max_times: 3
    retry_status_codes: [429, 502, 503, 504]
    disable_ssl_verification: false  # Not recommended for production

For more detailed information, you can refer to the <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/sink_docs/datahub.md|DataHub Rest Sink documentation> and the <https://datahubproject.io/docs/0.14.0/metadata-ingestion/cli-ingestion|CLI Ingestion documentation>.

Would you like more detailed steps or further assistance on any specific part of this configuration?

Sources:

3

Hi <@U086EAJ6RR8>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.