Troubleshooting Multiple Ingestion Triggers with Standalone MAE and MCE Consumers

ingestion
1

Original Slack Thread

Hi Team, I am facing an issue where I am seeing multiple Ingestion triggers happening for a scheduled ingestion sources. I am using standalone MAE and MCE consumers with 3 replicas. I am able to see 3 Ingestions getting triggered at each scheduled interval. can someone help me to resolve this issue? Thanks
I am using datahub version v0.12.0

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

With standalone MAE and MCE consumers, the MCE consumer should only have a single replica, otherwise each of the MCE consumers will create the corresponding aspects for the ingestion (for which the datahub-actions pipeline is listing for, to start the ingestion): https://github.com/acryldata/datahub-helm/blob/89c92c8ac73b4dc371d647216d60dff28cc7c9ae/charts/datahub/subcharts/datahub-mce-consumer/templates/deployment.yaml#L11-L12|https://github.com/acryldata/datahub-helm/blob/89c92c8ac73b4dc371d647216d60dff28cc[…]atahub/subcharts/datahub-mce-consumer/templates/deployment.yaml

4

okay, thanks for the confirmation.