Resolving DataHub Ingestion Error: Unrecognized Field `customProperties`

ingestion
1

Original Slack Thread

We’ve run into an ingestion issue that’s raising the following error:

    '/customProperties :: unrecognized field found but not allowed\n',
'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
    'message': 'Failed to validate record with class com.linkedin.domain.DomainProperties: ERROR :: /customProperties :: '
        'unrecognized field found but not allowed\n',
    'status': 422,
    'urn': 'urn:li:domain:***'}},```
This appears to be similar to <https://datahubspace.slack.com/archives/C029A3M079U/p1721224520731789|another user's unresolved issue>

gms version is `v0.13.3`
datahub cli version `0.13.3.4`

I believe this error has arisen after <https://github.com/ministryofjustice/data-catalogue/commit/0e4a234dbf634cfb4aa0136f30f6505b9f59fe29|upgrading the cli version to >`0.13.3.4`
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: We’re experimenting with a new AI assistant for questions in the <#CUMUWQU66> channel to help you track down helpful resources more quickly. Keep in mind that responses may not always be spot-on; feel free to ask it follow-up questions and don’t forget to provide its responses a :thumbsup: or :thumbsdown: so other folks in the Community know if you’re still looking for help on the topic!

3

To help you more effectively, could you please provide the following information:

  1. Are you using the UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g., 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g., BigQuery)

Based on the error message and the provided context, it seems like the issue is related to a field (customProperties) that is not recognized by the DataHub GMS. This is similar to other issues where unrecognized fields cause validation errors.

According to the <Issues · datahub-project/datahub · GitHub issue #9799>, such errors often occur due to version mismatches or unrecognized fields in the metadata being ingested. The error message indicates that the customProperties field is not allowed in the DomainProperties class.

To resolve this issue, you can try the following steps:

  1. Ensure Version Compatibility: Make sure that both your DataHub CLI and GMS versions are compatible. Sometimes, fields introduced in newer versions of the CLI may not be recognized by an older GMS version.
  2. Check for Field Validity: Verify that the customProperties field is valid and supported in the version of DataHub you are using. If it is not, you may need to remove or modify this field in your ingestion configuration.

Would you like more detailed steps on how to check for version compatibility or modify your ingestion configuration?

Sources:

4

It appears that <datahub/metadata-models/src/main/pegasus/com/linkedin/domain/DomainProperties.pdl at 56bb4c81113bf4fce8a48526080375480d619ba8 · datahub-project/datahub · GitHub should support customProperties> (and it did with the previous cli version)