Troubleshooting Looker-Snowflake File Lineage Ingestion and Visibility

ingestion
1

Original Slack Thread

Hi DH Team,
I am trying to create a file lineage between my looker and Snowflake. Following is the lineage file

- entity:
    env: PROD
    name: thirty_madison_snowflake.view.real_time_benefits
    platform: looker
    type: dataset
    platform_instance:
  fineGrainedLineages:
  upstream:
  - entity:
      env: PROD
      name: analytics.dbt_prod_base.real_time_benefits
      platform: snowflake
      type: dataset
      platform_instance:
    upstream:
    fineGrainedLineages:
version: '1'```
Following is the file that does the ingestion
```source:
  type: datahub-lineage-file
  config:
    file: ./looker_lineage.yml
    preserve_upstream: False```
When i am executing the command I can see the following output:
```bash-4.2# datahub ingest -c looker_feed_lineage_recepie.dbhub.yaml
[2024-04-29 20:51:01,544] INFO     {datahub.cli.ingest_cli:147} - DataHub CLI version: 0.13.0
[2024-04-29 20:51:01,601] INFO     {datahub.ingestion.run.pipeline:238} - Sink configured successfully. DataHubRestEmitter: configured to talk to <http://datahub-datahub-gms.datahub.svc.cluster.local:8080> with token: eyJh**********_H-8
[2024-04-29 20:51:01,695] INFO     {datahub.ingestion.run.pipeline:255} - Source configured successfully.
[2024-04-29 20:51:01,696] INFO     {datahub.cli.ingest_cli:128} - Starting metadata ingestion
-/var/lang/lib/python3.9/site-packages/pydantic/_internal/_decorators_v1.py:155: ConfigurationWarning: env is deprecated and will be removed in a future release. Please use platform_instance instead.
  return validator(values)
[2024-04-29 20:51:01,721] INFO     {datahub.ingestion.source.metadata.lineage:197} - Upstream detected for env='PROD' name='thirty_madison_snowflake.view.real_time_benefits' type='dataset' platform='looker' platform_instance=None. Extracting urn...
[2024-04-29 20:51:01,848] INFO     {datahub.cli.ingest_cli:141} - Finished metadata ingestion
/
Cli report:
{'cli_version': '0.13.0',
 'cli_entry_location': '/var/lang/lib/python3.9/site-packages/datahub/__init__.py',
 'py_version': '3.9.19 (main, Apr  9 2024, 12:36:07) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-17)]',
 'py_exec_path': '/var/lang/bin/python3',
 'os_details': 'Linux-5.10.205-195.807.amzn2.x86_64-x86_64-with-glibc2.26',
 'mem_info': '71.1 MB',
 'peak_memory_usage': '71.1 MB',
 'disk_info': {'total': '21.46 GB', 'used': '12.34 GB', 'used_initally': '12.34 GB', 'free': '9.13 GB'},
 'peak_disk_usage': '12.34 GB',
 'thread_count': 2,
 'peak_thread_count': 2}
Source (datahub-lineage-file) report:
{'events_produced': 2,
 'events_produced_per_sec': 5,
 'entities': {'dataset': ['urn:li:dataset:(urn:li:dataPlatform:looker,thirty_madison_snowflake.view.real_time_benefits,PROD)']},
 'aspects': {'dataset': {'upstreamLineage': 1, 'status': 1}},
 'warnings': {},
 'failures': {},
 'start_time': '2024-04-29 20:51:01.695913 (now)',
 'running_time': '0.35 seconds'}
Sink (datahub-rest) report:
{'total_records_written': 2,
 'records_written_per_second': 4,
 'warnings': [],
 'failures': [],
 'start_time': '2024-04-29 20:51:01.594231 (now)',
 'current_time': '2024-04-29 20:51:02.043899 (now)',
 'total_duration_in_seconds': 0.45,
 'max_threads': 15,
 'gms_version': 'v0.13.0',
 'pending_requests': 0}

 Pipeline finished successfully; produced 2 events in 0.35 seconds.```
However, while looking for the lineage I do not see any lineage that got created in the UI. Can you please let me know what am I doing wrong here
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

• CLI and UI both. CLI version 0.13.0. GMS version 0.13.0
• Lineage File Ingestion

4

Alternately, please let me know whaty should be the datahub version compatibility for allowing file ingestion via datahub cli