Question about Databricks Connectors and Privileges: Accessing Legacy HMS with Unity Catalog Connector

getting-started
1

Original Slack Thread

Question about <https://datahubproject.io/docs/generated/ingestion/sources/databricks/|Databricks Connectors and Privileges>
I am supporting someone who has successfully connected to Databricks using the Hive recipe and is now trying to switch to the Unity Catalog one.
They have been able to access the Unity Catalog catalogs but they cannot access the old HMS.
They are using a Service Principle for authentication and they’ve given it the privileges mentioned in the doc.
They seem to think that giving the SP Metastore Administration privileges will help, but I really, really think that’d be a bad idea given the damage those privileges can do.

So:

  1. Should they be able to access the legacy HMS using the UC connector?
  2. If not, should they continue to use the old one?
  3. Would granting those extra privileges help?
    Thank you!

fyi <@U04J2SE14DV>

2

<@U02G4B6ADL6> please, can you help in this?

3

And for (2), I should have clarified: Should they use both connectors?

4

Hey <@U04JBAY4KAA> - we recently added more documentation regarding permissions required to ingest hive metastore using the existing “unity-catalog” aka “databricks” ingestion. https://datahubproject.io/docs/next/generated/ingestion/sources/databricks#:~:text=Privileges%20documentation-,To%20ingest%20legacy%20hive_metastore,-catalog%20(include_hive_metastore|https://datahubproject.io/docs/next/generated/ingestion/sources/databricks#:~:text=Privileges%20documentation-[…],-catalog%20(include_hive_metastore

Let me know if providing only the documented privileges helps solve the problem for you.

5

This looks promising! I don’t think it was clear that when using the UC connector, the HMS catalog required different permissions