Investigating Disk Space Issue with MetadataChangeLog_Versioned_v1 in Datahub环境

Original Slack Thread

Hi all, I am looking for advice on how to best investigate an issue we have with datahub in multiple environments. We are using datahub 0.10.5 deployed on k8s. In some of our environments, kafka started to run out of disk space. Disk usage seemed to be stable, but it started to increase suddenly, as you can see from the graph (the graph shows the available disk space, not disk usage).

Most of the disk space seems to be used by one particular topic MetadataChangeLog_Versioned_v1 . The topic only has a retention of 7days as per kafka default.
I am fairly new to Datahub but I couldn’t find anything that explains the sudden increase of messages in the topic.
Does anyone have an idea what might cause this or where should I look at? TIA

I am also facing the same issue…
I am using datahub 0.12.1 and helm chart 0.13.19

unfortunately we got no help on this issue. The only thing we did to mitigate the issue was to set a retention size on the topic to avoid using all the disk space.
But we don’t know what triggered this increase…

<@U067V13943C> In my case I’ve noticed that sometimes a scheduled ingestion run does not end. It keeps feeding the bulk request in a loop until I cancel the ingestion… This happened this weekend and it seems datahub produced more than 28k messages in MetadataChangelog_versioned_v1