Troubleshooting lineage tab issue in ingestion pipeline with DataHub via CLI

ingestion
1

Original Slack Thread

Hi there - I am having some headaches with an ingestion pipeline. My ingestion runs via CLI, here are the details:

• ingestion via cli, datahub --version acryl-databu, version 0.13.2.1
• datahub version: 0.13.2
• RDS, Postgres
• DataHub running locally via Kubernetes
My recipe:

    type: postgres
    config:
        host_port: 'xxx:5430'
        database: xxx
        username: xxx
        include_tables: true
        include_views: true
        include_table_location_lineage: true
        include_view_lineage: true
        profiling:
            enabled: true
            profile_table_level_only: true
        password: xxx

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"
    token: xxx```
The ingestion process runs fine. The problem is with my lineage: the tab is empty for all my dataset and I don't understand where the issue could be. I didn't see any error coming up in the ingestion logs and I can't really debug this further on my own. Is there anyone who has any idea?
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

My expectations would be: see relationship between tables via foreign keys constraint

4

Foreign key constraints are not lineage. Lineage comes from things like create view or create table as select or insert into statements

5

Ok thanks for clarifying this Harshal.