Understanding and Addressing the "CooperativeTimeout" Error in DataHub Ingestion

ingestion
1

Original Slack Thread

<@U06TM7M40RK>
[2025-04-22 16:13:16,969] INFO {datahub.ingestion.source.dbt.dbt_common:1290} - Failed to generate CLL for model.xxx.foo: CooperativeTimeout deadline exceeded what is this error?

2

It seems likely that the error message “Failed to generate CLL for model.xxx.foo: CooperativeTimeout deadline exceeded” is related to a timeout mechanism used within the DataHub ingestion process, specifically when dealing with dbt (data build tool) models. The error indicates that the process to generate Column Lineage (CLL) for a specific dbt model exceeded the allowed time limit.

Based on the https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/utilities/cooperative_timeout.py|cooperative_timeout.py file, the CooperativeTimeout is a mechanism designed to handle timeouts in a cooperative manner. It requires the code to periodically call a function (cooperate) to check if the timeout has been exceeded. If the deadline is exceeded, a CooperativeTimeoutError is raised.

This mechanism is useful in scenarios where you have control over the code and can modify it to regularly check for timeouts, which seems to be the case here with the dbt ingestion process.

Would you like more detailed steps on how to address this issue or further explanation on the cooperative timeout mechanism?

Sources:

3

<@U06TM7M40RK> how can i increase it?

4

To increase the timeout for the cooperative timeout mechanism in DataHub, you would need to adjust the timeout value specified when using the cooperative_timeout context manager. This is typically done in the code where the timeout is being applied.

Here’s a general approach to increase the timeout:

  1. Locate the Code: Identify where the cooperative_timeout is being used in the DataHub ingestion code, particularly in the dbt ingestion process.

  2. Adjust the Timeout Value: Modify the timeout value passed to the cooperative_timeout context manager. This value is specified in seconds.

For example, if you find a section of code like this:

with cooperative_timeout(timeout=30):  # 30 seconds
    # Code that requires a timeout

You can increase the timeout by changing the value:

with cooperative_timeout(timeout=60):  # 60 seconds
    # Code that requires a timeout
  1. Rebuild and Deploy: After making the changes, you would need to rebuild and deploy the DataHub ingestion service to apply the new timeout settings.

If you are not directly modifying the source code, you might need to check if there are configuration options available in your DataHub setup that allow you to specify the timeout value for dbt ingestion. This would typically be part of the ingestion configuration files or environment variables.

Would you like more detailed guidance on how to locate and modify the specific code or configuration in your setup?

Sources:

5

<@U06TM7M40RK> but its happening in the ingestion, why is it happening?

6

The “CooperativeTimeout deadline exceeded” error during the ingestion process indicates that a specific operation within the DataHub ingestion pipeline, particularly related to generating Column Level Lineage (CLL) for a dbt model, is taking longer than the allowed time limit. This can happen for several reasons:

  1. Complex Queries: The SQL queries or transformations involved in generating the lineage might be complex, leading to longer processing times.

  2. Large Data Volumes: If the dbt model involves large datasets or numerous dependencies, the processing time can increase significantly.

  3. Resource Constraints: The system running the ingestion process might be resource-constrained (e.g., limited CPU or memory), causing delays in processing.

  4. Inefficient SQL Parsing: The SQL parsing mechanism used to generate lineage might not be optimized for certain types of queries or structures, leading to timeouts.

  5. Configuration Issues: There might be configuration settings that are not optimized for your specific use case, such as insufficient timeout settings or incorrect resource allocations.

To address this issue, you can consider the following steps:

  • Optimize SQL Queries: Review and optimize the SQL queries or transformations involved in the dbt model to reduce complexity and execution time.

  • Increase Timeout: As previously mentioned, increase the timeout value for the cooperative timeout mechanism to allow more time for processing.

  • Allocate More Resources: Ensure that the system running the ingestion process has sufficient resources (CPU, memory) to handle the workload.

  • Review Configuration: Check the ingestion configuration for any settings that might be contributing to the issue and adjust them as needed.

Would you like more detailed steps on any of these points or further assistance in troubleshooting the issue?

Sources:

7

<@U06TM7M40RK>

is there a way to increase via the recepeis?

8

The cooperative timeout mechanism itself is not directly configurable via the ingestion recipes in DataHub. However, you can adjust other related configurations in the recipe that might help mitigate the timeout issue. Here are some steps you can take:

  1. Increase General Timeout Settings: While the cooperative timeout is not directly adjustable via recipes, you can increase other timeout settings that might indirectly affect the processing time. For example, if there are any timeout_sec parameters in your recipe, you can increase them to allow more time for the ingestion process.

  2. Optimize Resource Allocation: Ensure that your ingestion process has sufficient resources by adjusting the resource allocation in your environment. This might involve increasing the CPU or memory limits for the ingestion process.

  3. Adjust Batch Sizes: If your ingestion process involves batching, you can adjust the batch sizes to process smaller chunks of data at a time, which might help avoid timeouts.

  4. Review and Optimize SQL Parsing: If the timeout is related to SQL parsing, ensure that your SQL queries are optimized and that any unnecessary complexity is reduced.

  5. Check for Configuration Options: Review the DataHub documentation or your specific setup to see if there are any configuration options available for adjusting timeouts or resource limits.

If you have access to the source code or configuration files, you might need to make changes there to adjust the cooperative timeout specifically.

Would you like more detailed guidance on any of these steps or further assistance with your specific setup?

Sources:

9

<@U06TM7M40RK>
process_refinements
boolean
When enabled, looker refinement will be processed to adapt an existing view.
Default: False

what does this mean?

10

The process_refinements configuration option in DataHub’s Looker ingestion process is related to handling LookML refinements. When this option is enabled, it allows the ingestion process to apply LookML refinements to existing views. Here’s what it means:

  1. LookML Refinements: In Looker, refinements are a way to modify or extend existing views or explores without altering the original LookML files. This is useful for customizing views for specific use cases or environments.

  2. Adapting Existing Views: When process_refinements is set to True, the ingestion process will apply these refinements to the views being ingested. This means that any changes or extensions defined in the refinements will be incorporated into the metadata ingested into DataHub.

  3. Default Behavior: By default, this option is set to False, meaning that refinements are not processed unless explicitly enabled. This is likely to ensure that the ingestion process remains straightforward unless refinements are needed.

  4. Use Cases: Enabling this option is beneficial when you have LookML refinements that you want to be reflected in the metadata ingested into DataHub. It allows for a more accurate representation of the Looker environment as it is used in practice.

If you have specific refinements in your LookML that you want to be considered during ingestion, you should enable this option.

Would you like more detailed information on how to enable this option or examples of LookML refinements?

Sources:

11

<@U06TM7M40RK> then why does me linage wont work when there are process_refinements True?

12

The DataHub’s documentation I have ingested does not mention how to troubleshoot lineage issues specifically related to enabling process_refinements. When process_refinements is set to True, it should allow LookML refinements to be processed and incorporated into the lineage. If lineage is not working as expected, you might want to check if the refinements are correctly defined in your LookML files and ensure that there are no errors in the ingestion logs that could indicate issues with processing these refinements. If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :light_bulb: Hint: Mention @runllm in the thread for followups.

13

<@U06TM7M40RK> if i have a looker view that has a calculation, for example:
mobile_active_users
measure: mobile_active_users {
value_format_name: decimal_0
type: count_distinct
sql: ${user_id} ;;
filters: [mobile_logins: “>0”]
}

why wont i be able to see the lineage between the user id in the bigquery dataset and mobile_active_users?

14

The issue you’re encountering with lineage not being displayed between the user_id in the BigQuery dataset and the mobile_active_users measure in Looker could be due to several factors related to how lineage is generated and displayed in DataHub. Here are some potential reasons and considerations:

  1. SQL Parsing Limitations: The SQL parsing mechanism used by DataHub might not fully support the specific SQL syntax or constructs used in your LookML. For example, if the SQL expression involves complex logic or unsupported functions, it might not be parsed correctly to generate lineage.

  2. Column-Level Lineage: DataHub supports column-level lineage, but it requires that the SQL expressions are parsed correctly to map columns from the source dataset to the Looker view. If the parsing fails or is incomplete, the lineage might not be established.

  3. Configuration Settings: Ensure that your DataHub ingestion configuration is set up to extract column-level lineage. This includes enabling options like extract_column_level_lineage in your Looker and LookML ingestion configurations.

  4. LookML Syntax: Verify that the LookML syntax is correct and that the sql parameter in the measure is properly referencing the user_id. Any syntax errors or misconfigurations could prevent the lineage from being established.

  5. Ingestion Logs: Check the ingestion logs for any warnings or errors related to SQL parsing or lineage extraction. These logs can provide insights into what might be going wrong during the ingestion process.

  6. DataHub Version: Ensure that you are using a version of DataHub that supports the features you need. There might be improvements or bug fixes in newer versions that address lineage issues.

If you would like more detailed guidance on troubleshooting this issue or specific steps to take, please let me know.

Sources:

15

<@U06TM7M40RK> IF I want a more detailed log? how can i achive that? while ingestion? is there a flag?

16

To achieve more detailed logging during the ingestion process in DataHub, you can enable debug logging. This can be done by setting the --debug flag when running the ingestion command. Here’s how you can do it:

  1. Using the CLI: When you run the ingestion command using the DataHub CLI, you can add the --debug flag to enable detailed logging. This will provide more verbose output, which can help in troubleshooting issues.

    datahub ingest -c your_recipe.yml --debug

  2. Environment Variable: You can also set the DATAHUB_DEBUG environment variable to true to enable debug logging globally. This can be useful if you want to enable debug logging for all DataHub operations.

    export DATAHUB_DEBUG=true

  3. Log File: If you want to capture the debug logs in a file for later analysis, you can redirect the output to a file:

    datahub ingest -c your_recipe.yml --debug &> /path/to/logfile.log

  4. Configuration Files: If you are using configuration files for ingestion, ensure that any logging-related settings are configured to capture debug-level logs.

By enabling debug logging, you will get more detailed information about the ingestion process, including any errors or warnings that occur. This can be particularly helpful for diagnosing issues with SQL parsing, lineage extraction, or other aspects of the ingestion process.

Would you like more detailed steps or further assistance with enabling debug logging?

Sources: