Is it safe to enable `use_queries_v2` in Snowflake ingestion? Why is it disabled by default?

ingestion
1

Original Slack Thread

Is it safe to enable use_queries_v2 in Snowflake ingestion? Why is it disabled by default?

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

It seems likely that enabling use_queries_v2 in Snowflake ingestion is generally safe and recommended for new users. According to the <https://datahubproject.io/docs/0.14.0/generated/ingestion/sources/snowflake#config-details|DataHub documentation>, use_queries_v2 is a configuration option that, when enabled, uses a new queries extractor to extract queries from Snowflake. This feature is not enabled by default primarily due to backward compatibility concerns, as mentioned in a <https://github.com/datahub-project/datahub/pull/12269|GitHub pull request>. The default setting is to ensure that existing users who rely on the previous behavior do not experience unexpected changes.

The new queries extractor likely offers improvements or additional features over the previous version, which is why it is recommended for new users or those setting up Snowflake ingestion for the first time. However, for users with existing setups, enabling this feature might require additional testing to ensure compatibility with their current configurations and workflows.

Would you like more detailed information on how to enable use_queries_v2 or any specific concerns you have about its impact on your ingestion process?

Sources:

4

What are the backward compatibility concerns?

5

<@U06TM7M40RK> What are the backward compatibility concerns?

6

It might be that the backward compatibility concerns related to enabling use_queries_v2 in Snowflake ingestion are primarily about ensuring that existing workflows and configurations continue to function as expected without disruption. According to the <https://github.com/datahub-project/datahub/pull/12269|GitHub pull request>, the feature is not enabled by default to avoid potential issues for users who have already set up their ingestion processes with the previous version of the queries extractor.

Some possible concerns could include:

  1. Changes in Data Extraction: The new queries extractor might change the way data is extracted or processed, which could lead to differences in the data ingested compared to the previous version.

  2. Configuration Adjustments: Users might need to adjust their configurations or ingestion recipes to accommodate the new extractor, which could be a barrier for those who have stable, existing setups.

  3. Testing and Validation: Enabling a new feature might require additional testing and validation to ensure that it integrates well with other components of the data ingestion pipeline and does not introduce new issues.

  4. Feature Parity: There might be features or behaviors in the old extractor that are not yet fully replicated in the new version, which could affect users relying on those specific functionalities.

Would you like more detailed information on how to address these concerns or any specific guidance on transitioning to use_queries_v2?

Sources:

7

<@U06TM7M40RK> yes I would

8

I’m not able to find any data sources to answer your question. You might want to specify which aspects of transitioning to use_queries_v2 you are most concerned about, such as configuration changes or testing procedures. If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.

9

<@U01GZEETMEZ> <@U01JUBAPZPS> can you give guidance on this? any reason this would not be safe to turn on?

10

Hey <@U04583XMT5Z> - we are going to be setting v2 as the default soon. Please go ahead and use.

11

excellent, thank you

12

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><