Understanding the purpose of datahub-mae-consumer and datahub-mce-consumer containers in a deployed Datahub on ECS

Original Slack Thread

hi there, ive deployed datahub on ecs having deployed 4 containers datahub-frontend, datahub-gms, datahub-actions and confluent-schema-registry, eveything seems to be working fine having ingested metadata from numerous sources. Im just a bit confused as to the purpose of the datahub-mae-consumer and datahub-mce-consumer components and whether i need to deploy these aswell as everything seems to be working fine as is

Hey Randal, those two containers are used for ingestion and handling certain types of writes in the product. Are you planning to use ingestion to continue adding data to your deployment or will it primarily be used in a read-only way?

currently we ingest data into datahub in two ways, firstly by the ui pointing it to snowflake and looker, secondly we run datahub ingest -c datahub/recipe.yml for dbt ingestion, moving forward we plan to use the same mechanisms

can you give a specific example where we would need those 2 other containers

Both those ingestions you mentioned will use the consumers!

but we havent deployed either of those containers :upside_down_face:

are they embedded in the gms or frontend containers?

<@U04N9PYJBEW> do you know about this one? When these would be needed?

I believe those are both embedded within GMS – they are <https://datahubproject.io/docs/metadata-jobs/|spring jobs> and gms is a spring application. <@U05CJD391ND> can you confirm?

^ has been confirmed internally by the team

ok thanks