Troubleshooting Custom Transformer Error in DataHub UI Ingestion

ingestion
1

Original Slack Thread

<@U06TM7M40RK> i wrote custom transformer and i ingested my recipe through CLI and everything ok bit when i started ingesting my recipe through UI i got error and it says “datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure transformers: No module named ‘dataset_description_addin’” how to fix it?

2

The error you’re encountering indicates that the custom transformer module dataset_description_addin is not found when running the ingestion through the DataHub UI. This is a known limitation with the current DataHub UI ingestion process.

According to the <Slack discussion>, it is currently not possible to install a custom transformer to the action pod where UI ingestion runs. This means that custom transformers are not supported in the DataHub UI ingestion process at this time.

Here are the key points from the discussion:

  • Custom transformers can be used with the DataHub CLI but not with the DataHub UI.
  • The action pod for UI ingestion does not support installing custom transformers.

For now, the recommended approach is to continue using the DataHub CLI for ingestion if you need to use custom transformers.

Would you like more detailed steps on how to ensure your custom transformer works with the CLI, or any other assistance?

Sources:

  • <Slack discussion on custom transformers> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
3

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: We’re experimenting with a new AI assistant for questions in the <#CUMUWQU66> channel to help you track down helpful resources more quickly. Keep in mind that responses may not always be spot-on; feel free to ask it follow-up questions and don’t forget to provide its responses a :thumbsup: or :thumbsdown: so other folks in the Community know if you’re still looking for help on the topic!

4

I’m not able to find any data sources to answer your question. Could you provide more details about the specific issue or question you have regarding DataHub? If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.