Configuring PostgreSQL Ingestion With Table Relationships in DataHub

Original Slack Thread

how can I ingestion relationship of table in database postgresql config?

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

To ingest table relationships in a PostgreSQL database using DataHub, you need to ensure that your ingestion configuration is set up to capture lineage, which includes relationships such as foreign key constraints. Here’s how you can do it:

  1. Enable Lineage in Configuration: Make sure your ingestion recipe includes the necessary configurations to capture table lineage. This involves setting the include_table_lineage parameter to true.

  2. Sample Configuration: Below is an example of a YAML configuration for ingesting PostgreSQL data with table relationships:

    source:
      type: postgres
      config:
        host_port: 'your_postgres_host:5432'
        database: 'your_database'
        username: 'your_username'
        password: 'your_password'
        include_tables: true
        include_views: true
        include_table_lineage: true  # Enable table lineage
        profiling:
          enabled: true
    sink:
      type: datahub-rest
      config:
        server: 'http://localhost:8080'
    
  3. Run the Ingestion: Use the DataHub CLI to run the ingestion with the above configuration.

    datahub ingest -c /path/to/your/ingestion_recipe.yaml
    
  4. Verify Lineage: After running the ingestion, you can verify the table relationships in the DataHub UI by navigating to the dataset page and checking the “Lineage” tab.

According to the <Postgres | DataHub Ingestion Documentation>, enabling include_table_lineage will allow DataHub to capture and display the relationships between tables, such as those defined by foreign keys.

Would you like more detailed steps or further assistance on any specific part of this process?

Sources:

  • <Postgres | DataHub Ingestion Documentation> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.

Hi, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><