Can DataHub Scan Cosmos DB in Azure for Spark Lineage?

ingestion
1

Original Slack Thread

<@U06TM7M40RK> can datahub scan cosmos db in azure?

2

It seems likely that DataHub does not currently support scanning or ingesting metadata directly from Azure Cosmos DB. According to a <https://github.com/datahub-project/datahub/issues/5169|GitHub issue>, there was a request for support for Azure Cosmos DB, but it appears that this feature is not yet implemented. Additionally, a <Slack discussion> also inquired about support for Cosmos DB, but there was no confirmation of support.

Would you like more detailed information or assistance on alternative approaches to integrate Cosmos DB with DataHub?

Sources:

3

<@U06TM7M40RK> so can we get spark lineage from a cosmos db?

4

It seems likely that obtaining Spark lineage from Azure Cosmos DB directly is not explicitly supported by DataHub. According to a <https://github.com/datahub-project/datahub/issues/5169|GitHub issue>, there was a request for support for Spark lineage with Azure Cosmos DB, but it appears that this feature is not yet implemented. The issue mentions that the AppendData command is not supported, which is relevant for Spark operations involving Cosmos DB.

If you are using Spark to read from or write to Cosmos DB, you might consider manually capturing lineage by emitting metadata using DataHub’s APIs or SDKs. This would involve capturing the input and output datasets in your Spark jobs and manually creating lineage entries in DataHub.

Would you like more detailed steps on how to manually capture and emit lineage for Spark jobs interacting with Cosmos DB?

Sources:

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><