Question about using dataHubExecutionRequest events for tracking ingestion completion statuses

Original Slack Thread

I am using 10.5 and working on an action to act after a successful ingestion. After checking the events that happen during an ingest, I saw that dataHubExecutionRequest type events have a task and status that can be used to check if it marks the end of an ingestion. When I try this approach on a OpenAPI ingestion, it has an event marking the end of the ingestion. However, when I try this approach on a mySQL ingestion, the last event the action receives has the ingest in a RUNNING status, even though the UI shows it as completed. Is this a known issue and/or is there known fix/workaround?

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!

Are you listening for the dataHubExecutionRequestResult⁣ aspect? If so, when the ingestion completes, you should see that aspect emitted with status⁣ equal to the status of that ingestion run (e.g. SUCCESS, FAILURE, or CANCELLED). I’m almost positive this is what the UI looks for when it displays status, so I would expect this to be emitted if the UI is updated.

Here are a couple logs to explain the issue. In the act method of the action, I’m checking if the entity is of entityType dataHubExecutionRequest and print that as well as task/status if it has one associated with it

attachment

The shorter log is the OpenApi ingest and ends in an event that shows it ended the ingest and contains all information as shown in the ingestion report of the UI. The longer log is the mySql ingest and ends with RUNNING even though the UI shows complete and I also queried that executionRequest using graphQL and it shows SUCCESS status as well.
<@U04N9PYJBEW>

Please also listen for the event dataHubExecutionRequestResult. I’m not sure why the OpenAPI ingestion produces the SUCCESS dataHubExecutionRequest but dataHubExecutionRequestResult is what you should really be listening for. Or perhaps both