Failed Validation of GlobalSettingsInfo Record with Unrecognized Field 'docPropagation'

ingestion
1

Original Slack Thread

<@U06TM7M40RK>
Failed to validate record {docPropagation={enabled=true, columnPropagationEnabled=false}, views={}} against its schema.
2024-12-10 06:22:17,655 [pool-4-thread-1] ERROR c.l.metadata.boot.BootstrapManager:40 - Caught exception while executing bootstrap step com.linkedin.metadata.boot.steps.IngestDefaultGlobalSettingsStep. Exiting…
com.linkedin.metadata.entity.validation.ValidationException: Failed to validate record with class com.linkedin.settings.global.GlobalSettingsInfo: ERROR :: /docPropagation :: unrecognized field found but not allowed

2

The error message you are encountering indicates that there is an unrecognized field docPropagation in the GlobalSettingsInfo record, which is not allowed according to the schema. This suggests that the field docPropagation is not defined in the GlobalSettingsInfo schema.

According to the <GlobalSettings | DataHub GlobalSettingsInfo documentation for version 0.13.1> and <GlobalSettings | DataHub 0.14.0>, the GlobalSettingsInfo schema does not include a field named docPropagation. This discrepancy is likely causing the validation error.

To resolve this issue, you should ensure that the configuration or data being ingested does not include the docPropagation field, or update the schema to include this field if it is intended to be part of the GlobalSettingsInfo.

Would you like more detailed guidance on how to modify the schema or configuration to address this issue?

Sources:

3

<@U06TM7M40RK>
give more details guidance

4

To address the validation error related to the docPropagation field in the GlobalSettingsInfo record, follow these detailed steps:

Step 1: Verify the Schema

  1. Check the Current Schema:

    • Locate the schema definition for GlobalSettingsInfo in your DataHub setup. This is typically found in the metadata-models module, specifically in a .pdl file such as GlobalSettingsInfo.pdl.
    • Ensure that the docPropagation field is defined in the schema. If it is not, this is likely the cause of the validation error.

  2. Update the Schema:

    • If the docPropagation field is intended to be part of the GlobalSettingsInfo, you will need to update the schema to include this field.

    • Here is an example of how you might define the docPropagation field in the GlobalSettingsInfo schema:

      @Record
record GlobalSettingsInfo {
  docPropagation: optional DocPropagationFeatureSettings
  // other fields...
}

    • Ensure that the DocPropagationFeatureSettings type is also defined in your schema.

Step 2: Rebuild and Redeploy DataHub

  1. Rebuild the Metadata Models:

    • After updating the schema, rebuild the metadata models to incorporate the changes. Run the following command from the root of the DataHub repository:

      ./gradlew :metadata-models:build

  2. Rebuild the Metadata Service:

    • Rebuild the metadata service to include the updated schema:

      ./gradlew :metadata-service:restli-servlet-impl:build

  3. Rebuild the Entire Project:

    • Rebuild the entire DataHub project to ensure all components are updated:

      ./gradlew build

  4. Deploy the Updated Metadata Service:

    • Deploy the updated metadata service (GMS) to your environment. This might involve updating your Docker images, Kubernetes deployments, or other deployment mechanisms.

      docker build -t your-docker-repo/datahub-gms:latest .
docker push your-docker-repo/datahub-gms:latest
5
  • Update your Kubernetes deployment or other orchestration tool to use the new image.

Step 3: Verify the Changes

  1. Check DataHub Logs:

    • Monitor the DataHub GMS and frontend logs to ensure that the services are running correctly and that the docPropagation field is being processed without errors.

  2. Verify Metadata in DataHub UI:

    • Check the DataHub UI to ensure that the docPropagation field is correctly displayed and searchable. This will confirm that the metadata is being ingested and processed as expected.

By following these steps, you should be able to resolve the validation error related to the docPropagation field in the GlobalSettingsInfo record. If you encounter any issues during this process, please let me know for further assistance.

Sources:

6

<@U06TM7M40RK>
What could be urn for globalsettinginfo ?

7

In DataHub, a URN (Uniform Resource Name) is used to uniquely identify resources. The general format for a URN in DataHub is:

urn:<Namespace>:<Entity Type>:<ID>

For the GlobalSettingsInfo entity, the URN would typically follow this structure. However, the specific details for GlobalSettingsInfo are not explicitly mentioned in the provided data sources. Based on the general pattern for URNs in DataHub, you can construct a URN for GlobalSettingsInfo as follows:

  • Namespace: This is usually li for LinkedIn, which is the default namespace used in DataHub.
  • Entity Type: This would be globalSettings or a similar identifier that represents the global settings entity.
  • ID: This is a unique identifier for the specific instance of the entity. For global settings, this might be a constant or a specific identifier like default.

An example URN for GlobalSettingsInfo might look like:

urn:li:globalSettings:default

This URN indicates that it is a global settings entity within the li namespace, and default is used as the identifier for the global settings instance.

If you need more detailed guidance or have specific requirements for constructing URNs, please let me know!

Sources:

8

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><