Creating Notifications for Metadata Changes Using DataHub Action Framework

ingestion
1

Original Slack Thread

I want to use action framework of Datahub Project. I need to create a notification after ingestion finished and new changes in metadata datasets and columns list should be on this notification. How can I do with action freamework. Sould I listen kafka? Which topic ?

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

To create a notification after ingestion finishes and to include new changes in metadata datasets and columns, you can leverage the DataHub Action Framework. Here’s a high-level overview of how you can achieve this:

  1. Use the Action Framework: The Action Framework allows you to listen to metadata changes and trigger actions based on those changes. You can listen to the MetadataChangeLog events to detect changes in datasets and columns.

  2. Set Up Notifications: You can configure notifications to be sent via Slack or email. According to the <https://github.com/datahub-project/datahub/blob/master/docs/managed-datahub/slack/saas-slack-setup.md|DataHub Slack Setup Guide>, you can set up Slack notifications for various events, including ingestion completion.

  3. Emit Events: After detecting changes, you can emit events back to DataHub or any other notification system you have configured. This can be done using the Action Framework to send notifications with the details of the changes.

  4. Configure the Action: You will need to configure an action in your DataHub instance to listen for the specific events you are interested in (e.g., changes in datasets and columns) and then trigger the notification.

For more detailed steps, you can refer to the <https://datahubproject.io/docs/0.14.0/act-on-metadata|DataHub Actions Documentation> which provides guidance on setting up and using the Action Framework.

Would you like more detailed steps on setting up the Action Framework or configuring notifications?

Sources:

4

Hi, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><