<@U06TM7M40RK> As time goes by, the database capacity will naturally increase due to data from previous versions. How should this be organized?
To manage the growth of your database due to accumulating metadata versions in DataHub, you can configure and enforce retention policies. DataHub supports different types of retention policies to help manage database size effectively:
- Indefinite Retention: Keeps all versions of metadata aspects.
- Version-based Retention: Retains only the latest N versions of metadata aspects.
- Time-based Retention: Retains versions ingested within the last N seconds.
According to the <https://datahubproject.io/docs/0.14.0/advanced/db-retention|DataHub documentation on database retention>, these policies can be applied at different levels, such as global, entity-level, or aspect-level. The retention policies are applied during GMS boot-up and on every ingest if an existing aspect is updated.
To configure these policies, you can create a YAML file specifying the retention settings and place it in the /etc/datahub/plugins/retention directory. For example, you can set a policy to keep only the latest 20 versions of all entity-aspect pairs:
- entity: "*"
aspect: "*"
config:
retention:
version:
maxVersions: 20
By restarting the DataHub GMS after creating the YAML file, the new retention policies will be applied.
Would you like more detailed guidance on setting up these retention policies or any other aspect of managing database growth in DataHub?
Sources:
- <https://datahubproject.io/docs/0.14.0/advanced/db-retention|Configuring Database Retention>
- <https://github.com/datahub-project/datahub/blob/master/docs/advanced/db-retention.md|Configuring Database Retention on GitHub>
0 button 
0 button 
Hint: Mention @runllm in the thread for followups.
Hi 
! Did this interaction resolve your issue or do you still require assistance ?
Please tag @ RunLLM if you still need further assistance !
Or tag @Datahub Community Support if you require a person to take a look
Hope this helps ! ><