Issue with Snowflake ingestion due to missing platform_instance value in foreign keys

ingestion
1

Original Slack Thread

Hi,

We are facing an issue with Snowflake ingestion : the urns of foreing keys do not match urn of existing datasets/fields because they are missing platform_instance value.

For instance, our platform_instance param is “my_platform” and the urn of our dataset is : “urn:li:dataset:(urn:li:dataPlatform:snowflake,my_platform.db_dev.test_dlk.dlk_fin_ref_silo,DEV)”
But the value of foreignDataset is : “urn:li:dataset:(urn:li:dataPlatform:snowflake,db_dev.test_dlk.dlk_fin_ref_silo,DEV)”
(Same issue for values of foreignFields : “urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:snowflake,db_dev.test_dlk.dlk_fin_ref_silo,DEV),sub_silo_en)”
This produce an “object not found error” when we click on the foreign key in the UI (which is expected, as urn does not exist.
Please note that the sourceFields values are correct as the URN contains the platform_instance value (“my_platform”), and the ingestion is performed correctly, except for these foreignDataset/foreignFields.

Here is An instance of database value after ingestion for aspect schemaMetadata and the selected urn :
{
“name”: “FKEY_DLK_REF_SILO_SILO_EN”,
“sourceFields”: [
“urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:snowflake,my_platform.db_dev.test_dlk.dlk_fin_silo_user_test,DEV),silo_en)”
],
“foreignFields”: [
“urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:snowflake,db_dev.test_dlk.dlk_fin_ref_silo,DEV),silo_en)”
],
“foreignDataset”: “urn:li:dataset:(urn:li:dataPlatform:snowflake,db_dev.test_dlk.dlk_fin_ref_silo,DEV)”
}

Please find attached the recipe used for ingestion

Is there any way we could add the platform_instance value from recipe to the urn of foreignKeys, so they match the exsiting URNS generated by the snowflake ingestion?

Any help would be grealty appreciated.
Thanks
Antoineattachment

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

Should be fixed by https://github.com/datahub-project/datahub/pull/9577

4

Thanks for the detailed bug report :slightly_smiling_face:

5

Hello <@U01GZEETMEZ> thank you very much for your reply and the quick fix it is greatly appreciated.

6

Hello <@U01GZEETMEZ> may I ask in which version this correction will be available, if you have an idea of the release date of this fix? Thanks again for your help.

7

It’s available in an rc release 0.12.1.4rc10, and also will be part of the next CLI release of acryl-datahub

8

<@U01GZEETMEZ> thank you so much for the information and the help, have a nice day