Debugging Failed Ingestion Source Creation Errors in DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> debug Failed to create ingestion source!: An unknown error occurred.

2

To debug the error message “Failed to create ingestion source!: An unknown error occurred,” you can follow these steps:

  1. Check Logs for Detailed Errors: Review the logs from the DataHub ingestion executor and the DataHub GMS to identify any specific error messages. This can provide insights into why the ingestion source creation is failing. You can access the logs by clicking on the Details button of the failed ingestion run in the UI. According to the <https://datahubproject.io/docs/0.14.0/ui-ingestion#debugging-a-failed-ingestion-run|DataHub documentation>, the output of each run is captured and available to view in the UI for easier debugging.

  2. Verify Configuration: Ensure that your ingestion recipe and configurations are correct. Sometimes, a misconfiguration can lead to such errors. Double-check fields like host_port, username, password, and other required fields to ensure they are correctly set.

  3. Network Connectivity: Make sure that the DataHub instance can reach the data source and that there are no network issues. Verify DNS resolution and network policies that might be blocking the connection.

  4. Container Health: Ensure that all necessary Docker containers are running and healthy. You can verify the health of the containers using docker ps. Specifically, check the datahub-actions container, which is responsible for executing ingestion requests.

  5. Resource Allocation: Ensure that your Docker environment has sufficient resources allocated to the DataHub components. According to a <Slack conversation>, resource exhaustion can lead to such errors.

  6. Version Compatibility: Verify that all components (e.g., DataHub backend, ingestion connectors) are compatible with your current DataHub version. Sometimes, upgrading to a newer version of DataHub can resolve issues.

Would you like more detailed steps or further assistance on any specific part of the process?

Sources:

3

<@U06TM7M40RK> this is a EKS based deployment that has an running snowflake ingestion does it stop from adding new sources ?

4

The DataHub’s documentation I have ingested does not mention how running a Snowflake ingestion on an EKS-based deployment might prevent adding new sources. It would be helpful to know if there are any specific error messages or logs when attempting to add new sources, as well as the current resource allocation and status of the DataHub components on your EKS cluster. If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :light_bulb: Hint: Mention @runllm in the thread for followups.

5

By the way, I found these links that might be relevant to what you’re looking for. Let me know if this helps: - UI Based Ingestion / Managed Ingestion - DataHub - Snowflake Ingestion Guide: Overview - DataHub - Datahub ingestion pipeline guide - Restack

6

<@U06TM7M40RK> debug EKs setup for debug Failed to create ingestion source!: An unknown error occurred.

7

To debug the “Failed to create ingestion source!: An unknown error occurred” issue in an EKS setup, you can follow these steps:

  1. Check Pod Logs: Start by checking the logs of the relevant pods in your EKS cluster. Specifically, look at the logs for the datahub-actions pod, which is responsible for executing ingestion requests. You can use the following command to view the logs:

    kubectl logs <datahub-actions-pod-name> -n <your-namespace>

    Replace <datahub-actions-pod-name> and <your-namespace> with the actual pod name and namespace.

  2. Verify Pod Status: Ensure that all DataHub-related pods are running and healthy. You can check the status of the pods using:

    kubectl get pods -n <your-namespace>

  3. Inspect Resource Allocation: Check if there are sufficient resources (CPU, memory) allocated to the DataHub components, especially the datahub-actions pod. Resource constraints can lead to failures in creating ingestion sources.

  4. Network Connectivity: Verify that the EKS cluster has network access to the Snowflake instance and any other data sources. Network issues can cause the connection to fail. Ensure that there are no firewall or DNS issues preventing connectivity.

  5. Configuration Verification: Double-check your ingestion configuration for any errors or missing fields. Ensure that all required fields like account_id, username, password, and role are correctly set in your Snowflake ingestion configuration.

  6. Check for Known Issues: There might be known issues with the version of DataHub or the ingestion connectors you are using. Check the <https://github.com/datahub-project/datahub/issues|DataHub GitHub issues> for any related issues.

  7. Review EKS Setup: Ensure that your EKS setup is correctly configured, including IAM roles, security groups, and network policies. According to the <Amazon EKS cluster IAM role - Amazon EKS EKS documentation>, proper IAM roles and policies are crucial for accessing AWS resources.

Would you like more detailed steps or further assistance on any specific part of the process?

Sources:

8

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><