UI Sidebar Bugs with Ingested Datasets Displaying Incorrectly

troubleshoot
1

Original Slack Thread

Hi guys, we have issues with what I believe to be the new UI sidebar bugs.
It shows with Kafka, Tableau and Snowflake datasets we ingested, in different ways.

For snowflake, main and only DB that ingestion sees is called PROD. In the sidebar, almost all items are listed under ‘PROD’ followed by schema names in the dropdown, however a few items are in ‘prod’ (lowercase), again followed by schema names in the dropdown. I checked URNs of two tables, both found in same schema. One is in ‘PROD’ in sidebar, other is in ‘prod’, however the URNs match completely (except for the table name part obviously).

For Kafka, we gave the ingestion instance name data-engineering, and in total there are lets say 200 topics ingested. In the sidebar, ‘data-engineering’ appears twice, one has a number 110 next to it, and the other number 90, so for some reason they are split. The 110 group has everything within, there are no more dropdowns aka ‘folders’, while the 90 group has multiple folders within it, further separating the topics into folders based on name (dots in name are used as separator)

For Tableau, sidebar looks good, but when I tested it for a specific example (clicking subfolder that should contain 7 charts), I only find 5 charts in the list. Oddly enough, I can find the missing chart directly, and also if I directly search by name for the folder where the charts should reside, I can open it and find all 7 charts there.

Has anyone else faced some these issues, are there fixes planned?

2

<@U03BEML16LB> might be able to speak to this! Thanks for flagging this :slight_smile:

3

hey there! okay so the first issue here seems like a capitalization issue potentially. you said urns of the two different entities under the different folders have the same casing of “prod”? this has to do with how we’re producing filters based on this “origin” and that’s usually based on the entity urn or the origin aspect

the kafka situation could be because one of the data-engineering groups is based on an entity and another is not, so the entities inside of one got their browsePathsV2 aspect in a different way than the others. Are you ingesting data with an updated CLI version? which version? and did you run the browse backfill step when you originally upgraded to show the browse sidebar?

for the tableau case, does the sidebar say there are 7 things but the search results only show 5 when you click into it? and what do you mean you can open up the folder and find all 7 charts in there?

biggest thing here is wanting to ensure your ingestion CLI version is up to date and you’re running ingestion regularly to make sure the browsePathsV2 aspects are up to date - these aspects are what drive the sidebar

4

Hi Chris, thank you for response, I’ll try to explain myself a bit clearer.
We’re using latest major release (0.11.0) and CLI of ingestion job was also 0.11.0, checked in logs. Additionally, this is a fresh deployment, there was no upgrade. Finally, these issues above appeared since first ingestion job and persisted since.

For Snowflake, imagine two tables, found inside Snowflake DB called PROD, and schema SCHEMA1

PROD.SCHEMA1.TABLE2```
Their urns are exactly as you expect them, only different in table name part, however sidebar browse path looks like this
```Snowflake -&gt; PROD -&gt; SCHEMA1 -&gt; TABLE1
Snowflake -&gt; prod -&gt; SCHEMA1 -&gt; TABLE2```
For tableau example, I meant that if you search name of container(folder) that contains 7 charts, it finds it and open a list of 7 charts. However navigating to it in sidebar and clicking it only shows 5 charts in the list.

For Kafka, I'm not sure I understand what you mean by based on entity. There is nothing special between topics in one group vs the other, sidebar shouldn't be splitting them

Tagging for visibility <@U03BEML16LB>
5

Any chance you looked into this? <@U03BEML16LB>

6

hey i’m so sorry <@U05H287EB51> this has gotten lost on my radar until now. I’m going to keep looking into this but also tag <@U04N9PYJBEW> here as well from our ingestion team since at least two of these three issues have to do with how we’re generating our browsePathsV2 aspects. The kafka situation could be due to a race condition that we’ve seen a few times where some of these assets are receiving a default browsePathsV2 aspect while others are receiving what we get back in ingestion.

I’m unsure why a few tableau assets aren’t showing up in the sidebar. For this, would you be able to find the values of the browsePathsV2 aspects for these two missing tableau entities?

7

Can you post your recipe?

8

Recipes are out of the box pretty much, example for Snowflake without auth stuff:

    type: snowflake
    config:
        include_table_lineage: true
        include_view_lineage: true
        include_tables: true
        include_views: true
        profiling:
            enabled: false
            profile_table_level_only: true
        stateful_ingestion:
            enabled: true
        include_view_column_lineage: true```
9

Hmm, I really don’t know where the lowercase prod is coming from, especially if you say the urns are identical, i.e. it’s PROD in the urn