Using Custom Queries in Great Expectations (GX) for DataHub Integration

[user-2]

Original Slack Thread

Hello team,
My question related to great expectations
just wanted to know whether GX supports custom query (where clause)? Example: select * from table_name where organigation = ‘apple’ and created_date = ‘2025-02-05’

I want to validate the assertions with the above query
Cc <@U054NUV9D9T>

[user-2]

<@U06TM7M40RK>

[user-1]

Great Expectations (GX) itself does support filtering data using custom queries or “batch_kwargs” that include a SQL query or a WHERE clause, depending on the execution engine and how you configure your expectation suite. However, your question is about using such custom queries in the context of DataHub’s integration with Great Expectations.

DataHub’s GX Integration:

• DataHub’s integration with Great Expectations currently supports only SQL-based datasources using the SqlAlchemyExecutionEngine (<https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/integration_docs/great-expectations.md|see docs>).
• You can define expectations in GX that use custom queries or filters (e.g., using a WHERE clause) as long as your datasource is supported by SqlAlchemyExecutionEngine.
• For example, you can configure a batch in GX with a custom SQL query that includes your WHERE clause, and then run expectations on that batch.

Limitations:

• DataHub’s GX integration does not support non-SQLAlchemy execution engines (such as Pandas or Spark) out of the box (<Slack Slack discussion>, https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/integration_docs/great-expectations.md|docs).
• If you are using a supported SQL database (like Postgres, MySQL, etc.), you can use custom queries in your GX checkpoint, and DataHub will ingest the results as assertions.

Summary:
Yes, you can use custom queries (with WHERE clauses) in Great Expectations when using SQLAlchemyExecutionEngine, and DataHub will ingest the results if you configure the integration as described in the <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/integration_docs/great-expectations.md|official documentation>. For other engines or more advanced use cases, this is not currently supported out of the box.

Would you like more detail on how to set up such a custom query in your GX configuration for DataHub?

Sources:

• <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/integration_docs/great-expectations.md|see docs>
• <Slack Slack discussion> 0 button 0 button Hint: Mention @runllm in the thread for followups.