To deny specific schema patterns in your DataHub ingestion pipeline, you can modify the schema_pattern configuration by adding the deny field with the desired patterns. According to the <SQLAlchemy | DataHub documentation for SQLAlchemy ingestion>, you can specify regex patterns to exclude certain schemas from ingestion.
Here’s how you can update your configuration to deny the specified schema patterns:
This configuration will exclude schemas named INTR, CDR, DRIMRT, REL, and AUDIT from being ingested. Would you like more details on any specific part of this configuration?
Sources:
<SQLAlchemy | DataHub SQLAlchemy Ingestion Configuration> 0 button 0 button Hint: Mention @runllm in the thread for followups.
To configure the table_pattern and view_pattern in your DataHub ingestion pipeline, you can use regex patterns to specify which tables and views to include or exclude. According to the <SQLAlchemy | DataHub documentation for SQLAlchemy ingestion>, you can use the allow and deny fields within these patterns to control the inclusion and exclusion of tables and views.
If you want to deny specific tables and views based on the schema patterns you mentioned earlier, you can set the deny field in both table_pattern and view_pattern to match the tables and views within those schemas. Here’s how you can update your configuration:
To configure schema_pattern, table_pattern, and view_pattern in your DataHub ingestion pipeline, you can use regex patterns to specify which schemas, tables, and views to include or exclude. These patterns are part of the configuration that allows you to filter the data being ingested based on your requirements.
schema_pattern
Purpose: To filter which schemas are included or excluded during ingestion.
Configuration: You can use allow and deny fields to specify regex patterns.