Addressing the Issue of Databricks Workspace Prefix Not Applying to Datahub Tables

ingestion
1

Original Slack Thread

<@U06TM7M40RK> Prefix of databricks workspace not applying on Datahub tables after ingestion

2

To address the issue of the Databricks workspace prefix not being applied to DataHub tables after ingestion, it is important to understand how platform instances are handled during the ingestion process. Based on the <https://github.com/datahub-project/datahub/pull/8671|GitHub pull request>, there was a known issue where the platform instance aspect was not being emitted correctly for datasets in Databricks ingestion. This could lead to inconsistencies in how datasets are represented in DataHub.

Here are some steps and considerations to resolve this issue:

  1. Platform Instance Configuration: Ensure that your ingestion configuration includes the correct platform_instance setting. This setting should be consistent with how other data sources are configured.

  2. Flag for Platform Instance Aspect: According to the pull request, there was a suggestion to add a flag to control the creation of the platform instance aspect. This flag should be set to ensure that the platform instance is correctly applied.

  3. Update DataHub Version: If you are using an older version of DataHub, consider updating to a newer version where this issue might have been addressed. The pull request indicates that changes were made to handle this aspect more consistently.

  4. Review Ingestion Logs: Check the ingestion logs for any warnings or errors related to the platform instance. This can provide insights into whether the configuration is being applied correctly.

Would you like more detailed steps on how to configure the platform instance or further assistance on a specific part of the process?

Sources:

3

Hi <@U07RXL145D0>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

4

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><