Hiding Partition Tables in PostgreSQL and DataHub Configurations

ingestion
1

Original Slack Thread

do y’all have an option to hide partition tables in postgres?

i have a table in my pg database that partitions by month and there are over 100 ‘extra’ tables

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

To hide or exclude partition tables in PostgreSQL from being ingested or displayed, you can use the table_pattern.deny configuration in DataHub. This allows you to specify a pattern to match the partition tables you want to exclude.

For example, if your partition tables follow a naming convention like table_name_pYYYYMMDD, you can use a regex pattern to deny these tables. According to a <Slack discussion>, a regex pattern such as .*_p\d+$ can be used to match and exclude tables that end with _p followed by digits.

Here’s a sample configuration snippet:

table_pattern:
  deny:
    - '.*_p\d+$'

This configuration will prevent DataHub from ingesting or displaying any tables that match the specified pattern.

Would you like more details on how to implement this or any other aspect of DataHub?

Sources:

  • <Slack discussion> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
4

if the table pattern is table_name_pYYYY_MM_DD, how do i change the deny pattern?

5

<@U06TM7M40RK>

6

If your partition tables follow the naming convention table_name_pYYYY_MM_DD, you can adjust the deny pattern to match this format. The pattern should account for the underscores between the year, month, and day.

Based on the <Slack discussion>, you can modify the regex pattern to:

table_pattern:
  deny:
    - '.*_p\d{4}_\d{2}_\d{2}$'

This pattern will match any table name that ends with _p followed by a four-digit year, an underscore, a two-digit month, another underscore, and a two-digit day.

Would you like more detailed steps on how to apply this pattern in your DataHub configuration?

Sources:

  • <Slack discussion> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
7

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><