How to Enable and Structure Emissions as Runs in DataHub

getting-started
1

Original Slack Thread

i am working with tasks and data jobs, and I notice the “runs” tab is greyed out. when i submit multiple jobs with the same urn, the information updates, but I would prefer to add these as runs. Could somebody point me towards the documentation which would help explain how to enable this tab and structure my emissions to indicate to datahub that i am using runs?

2

Here is an example python code about how you can create runs ->
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_job_dataflow_new_api_simple.py

3

Thank you! Is it possible to emit a job as an MCE work unit or only through calling this class’ emit method?

4

it just makes the call easier

5

it generates mcps

6

if you want the raw mcp you can get it easily as well

7

https://github.com/datahub-project/datahub/blob/8efe30f5f810bcbb7ce2b2f4e18fa4e3c5ca0f46/metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L214|https://github.com/datahub-project/datahub/blob/8efe30f5f810bcbb7ce2b2f4e18fa4e3c5[…]on/src/datahub/api/entities/dataprocess/dataprocess_instance.py

8

i see, I only ask since I have my connection configured as a DataHubGraph w/ a DatahubClientConfig so setting the rest emitter as emitter = DatahubRestEmitter(<<URL>>) doesn’t work as well unless there is some way to generate an emitter object from the graph which I am unaware of (entirely possible)

9

sure, it is up to you how you want to use :slightly_smiling_face:

10

thanks for your help!