Troubleshooting DataHub Ingestion Error: Union Type Not Backed by DataMap

ingestion
1

Original Slack Thread

can anyone help

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: We’re experimenting with a new AI assistant for questions in the <#CUMUWQU66> channel to help you track down helpful resources more quickly. Keep in mind that responses may not always be spot-on; feel free to ask it follow-up questions and don’t forget to provide its responses a :thumbsup: or :thumbsdown: so other folks in the Community know if you’re still looking for help on the topic!

3

I’m not able to find any data sources to answer your question. Could you provide more details about the specific issue or question you have regarding DataHub? If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.

4

fields/1/type/type :: union type is not backed by a DataMap

5

Hi <@U07CL4YP02D> to have RunLLM continue generation you will need to tag <@U06TM7M40RK> in follow up messages

hope this helps :slightly_smiling_face:

6

It looks like you’re encountering an error related to union types not being backed by a DataMap when ingesting metadata using the Java SDK. This issue can arise when the schema definition includes union types that are not properly handled.

Based on the provided data sources, here are some steps and considerations to address this issue:

  1. Schema Definition:
    Ensure that your schema definition correctly specifies the union types. Union types in DataHub need to be properly defined and backed by a DataMap.

  2. Validation Errors:
    The error message indicates that the union type is not backed by a DataMap or is null. This suggests that the schema might be missing necessary information or that the union type is not correctly defined.

  3. Example Schema:
    Here is an example of how a union type might be defined in a schema:

    {
  "type": "record",
  "name": "ExampleRecord",
  "fields": [
    {
      "name": "exampleField",
      "type": [
        "null",
        "string",
        {
          "type": "record",
          "name": "NestedRecord",
          "fields": [
            {
              "name": "nestedField",
              "type": "string"
            }
          ]
        }
      ]
    }
  ]
}

  4. Updating Snapshot Union:
    If you are creating a custom entity, ensure that you have added the Snapshot class to the Snapshot union. This is necessary for the entity to be recognized correctly. Refer to the <No Code Metadata | DataHub documentation> for more details.

  5. Common Issues:

    • Missing Fields: Ensure that all required fields are present in the schema.
    • Correct Data Types: Verify that the data types used in the schema are supported and correctly defined.

Would you like more detailed guidance on any of these steps or further troubleshooting tips?

Sources: