"Troubleshooting Custom Lineage Emission for MlModelGroup and MlFeatureTable"

troubleshoot
1

Original Slack Thread

Hi everyone!!

I’m trying to emit a custom lineage between an MlModelGroup and a MlFeatureTable, but I’m facing some problems when executing the script

'Unable to emit metadata to DataHub GMS: java.lang.RuntimeException: Unknown aspect upstreamLineage for entity mlmodelgroup'

What I’m doing wrong?
[In the thread there is the full script and traceback]

Thank you!

2 
from datahub.emitter.mcp import MetadataChangeProposalWrapper
import datahub.metadata.schema_classes as models
import datahub.emitter.mce_builder as builder
from datahub.emitter.rest_emitter import DatahubRestEmitter

def emit_lineage(entity:str, upstream_entities: List[str], emitter: DatahubRestEmitter):
    upstreams = [models.UpstreamClass(dataset=d, type=models.DatasetLineageTypeClass.TRANSFORMED) for d in upstream_entities]
    mcpw = MetadataChangeProposalWrapper("mlmodel_group", 
                                         models.ChangeTypeClass.UPSERT, 
                                         entityUrn=entity, 
                                         aspectName="upstreamLineage",
                                         aspect=models.UpstreamLineageClass(upstreams=upstreams))
    emitter.emit_mcp(mcp=mcpw)

if __name__ == "__main__":
    emitter = DatahubRestEmitter(gms_server="<http://localhost:8080>", token=None)

    model_entity = builder.make_ml_model_group_urn(platform="sagemaker", 
                                                   group_name="mlplatform-weight-size-prediction-prod", 
                                                   env="PROD")
    upstream_entities = [
        builder.make_ml_feature_table_urn(platform="feast", feature_table_name="user_features"),
    ]
    emit_lineage(model_entity, 
                 upstream_entities,
                 emitter)
    ```
3 
  File "/Users/ricardohortelano/miniconda3/lib/python3.10/site-packages/datahub/emitter/rest_emitter.py", line 266, in _emit_generic
    response.raise_for_status()
  File "/Users/ricardohortelano/miniconda3/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Server Error for url: <http://localhost:8080/aspects?action=ingestProposal>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ricardohortelano/kk/datahub/emit_custom_lineage.py", line 25, in &lt;module&gt;
    emit_lineage(model_entity, 
  File "/Users/ricardohortelano/kk/datahub/emit_custom_lineage.py", line 14, in emit_lineage
    emitter.emit_mcp(mcp=mcpw)
  File "/Users/ricardohortelano/miniconda3/lib/python3.10/site-packages/datahub/emitter/rest_emitter.py", line 245, in emit_mcp
    self._emit_generic(url, payload)
  File "/Users/ricardohortelano/miniconda3/lib/python3.10/site-packages/datahub/emitter/rest_emitter.py", line 274, in _emit_generic
    raise OperationalError(
datahub.configuration.common.OperationalError: ('Unable to emit metadata to DataHub GMS: java.lang.RuntimeException: Unknown aspect upstreamLineage for entity mlmodelgroup', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'message': 'java.lang.RuntimeException: Unknown aspect upstreamLineage for entity mlmodelgroup', 'status': 500})```
4

<@U01H9NETLCQ> Would love your help here!