Troubleshooting Lineage Generation for PostgreSQL Database in DataHub

ingestion
1

Original Slack Thread

Hello Folks,
we have tried to create lineage for postgres DB but after ingestion I have to manually create the lineage between the table. May i know what could be the reason. Can I get the lineage view automatically.

Thanks in advanced.
Divyam Goyal

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

To automatically generate lineage for a PostgreSQL database in DataHub, you need to ensure that your ingestion configuration is set up correctly. Here are some steps and considerations based on the available data sources:

  1. Configuration Settings: Make sure that your ingestion recipe includes the necessary configurations for capturing lineage. Specifically, you need to enable include_view_lineage if you want to capture lineage from views. Here’s an example configuration:

    source:
    type: postgres
    config:
        host_port: 'your_host:5432'
        database: 'your_database'
        username: 'your_username'
        password: 'your_password'
        include_tables: true
        include_views: true
        include_view_lineage: true
        profiling:
            enabled: true

    According to the <Postgres | DataHub Ingestion Documentation>, these settings are crucial for enabling lineage capture.

  2. Check for Errors: Review the logs for any errors or warnings that might indicate why lineage is not being captured. Logs can provide insights into configuration issues or other problems during ingestion.

  3. Version Compatibility: Ensure that the DataHub version you are using supports the features you need. Some features might not be available in older versions. As per the <https://datahubproject.io/docs/0.14.0/generated/lineage/lineage-feature-guide|DataHub Lineage Guide>, automatic lineage extraction is supported for various data tools, including PostgreSQL.

  4. Schema and Table Patterns: Verify that your schema and table patterns are correctly configured to include the tables and views for which you want to capture lineage.

If you have followed these steps and are still experiencing issues, it might be helpful to provide specific logs or error messages for further troubleshooting. Would you like more detailed steps or have specific logs/errors to share? This will help in providing a more targeted solution.

Sources:

4

Hi, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><

6

Hello <@U07FV3UHE4D> , can someone please help me with my question

7

Thank you for letting us know you need a person to take a look at this thread.

Either a community member or one of our team members will try to assist when they have a moment, responses unfortunately might not be immediate. We hope this helps speed things along :slight_smile: