Troubleshooting Anomalo & Datahub Integration Error with AssertionRunEventClass

ingestion
1

Original Slack Thread

Has anyone successfully gotten the Anomalo >> Datahub integration successfully working? We received some scripts from Anomalo
<@U04LXE8H2BB>

DataHub CLI version: 0.11.0.4

  • It appears to be working for some tables but then we received an error
     76 check_section_mcp = EmitMCP(i['run_config']['_metadata']['check_message'], dhtbl, emitter, assertion_dataPlatformInstance, platform)
     77 check_section_mcp.run()
---&gt; 78 assertion_pass_fail(i['run_config']['_metadata']['check_message'], dhtbl,"pass" if (i['results']['success'] == True) else "fail", i['check_run_url'], i['results']['evaluated_message'], platform)


AvroTypeException: The datum AssertionRunEventClass({'timestampMillis': 1698357760971, 'runId': '5ac580f6f88741d4a32813ff592a1316', 'asserteeUrn': 'urn:li:dataset:(urn:li:dataPlatform:bigquery,dbc-895f1218-7031.anomalo_unity.main.public.weather_stage,PROD)', 'status': 'COMPLETE', 'result': AssertionResultClass({'type': 'FAILURE', 'rowCount': None, 'missingCount': None, 'unexpectedCount': None, 'actualAggValue': None, 'nativeResults': {'URL': '<https://app.anomalo.com/dashboard/check_runs/9436355>', 'Msg': None}, 'externalUrl': '<https://app.anomalo.com/dashboard/check_runs/9436355>', 'error': None}), 'runtimeContext': None, 'batchSpec': None, 'assertionUrn': 'urn:li:assertion:b6c0bdedbe27ce84bd84cbbb5ea86e82', 'eventGranularity': None, 'partitionSpec': PartitionSpecClass({'type': 'FULL_TABLE', 'partition': 'FULL_TABLE_SNAPSHOT', 'timePartition': None}), 'messageId': None}) is not an example of the schema {
  "type": "record",
  "Aspect": {
    "name": "assertionRunEvent",
    "type": "timeseries"
[... much longer stacktrace]```
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

Hey Scott! I just came across this one and noticed there wasn’t a resolution… if you’re still hitting this issue, I highly suggest joining us at the Community Marathon tomorrow! We have a variety of troubleshooting & support sessions scheduled throughout the day where we can give live help :slight_smile: https://datahubspace.slack.com/archives/CUMV92XRQ/p1705433341623299

4

<@U04LXE8H2BB> - maybe we can join this marathon tomorrow. <@U0121TRV0FL> - do you have a suggested time we should join to take advantage of the troubleshooting session

5

<@U04LXE8H2BB> looks like the issue is that the AssertionResultClass.nativeResults expects a pure dict mapping strings -> strings, but there’s a value of "Msg": None

6

<@U01GZEETMEZ> Thanks for that. So if I’m understanding correctly, we need to understand why Anomalo is trying to push a value of None to this (and likely other) fields?

7

You likely just need to look through the code wherever nativeResults is set, and add str(...) around each value in the dict

8

Most of the other None values are fine

9

<@U06GHEQ75FT> ^^