Ingestion and Status Aspect Creation in DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> does ingestion automatically create status aspect for an entity

2

Ingestion does not always automatically create a status aspect for an entity. The behavior depends on the ingestion connector and the specific implementation. Here are the key points:

  • For some ingestion sources, DataHub has implemented a feature called “stateful ingestion,” which can automatically manage the status aspect (including soft deletion by setting removed: true when an entity is no longer present in the source). However, this is not universal for all connectors and must be enabled/configured where supported. For example, the DBT integration supports stateful ingestion, and you can enable it to have entities automatically soft-deleted when they are removed from the source (<Stateful Ingestion | DataHub docs>).

  • If stateful ingestion is not enabled or not supported for a particular source, the status aspect may not be created or updated automatically. In such cases, entities that are no longer present in the source may remain visible in DataHub until manually deleted or updated.

  • The status aspect is used to indicate whether an entity is “removed” (soft-deleted). An entity is visible in the DataHub UI if either the status aspect does not exist, or it exists and has removed: false (<Slack Slack discussion>).

  • There is also a utility in the ingestion framework (auto_status_aspect) that can add a status aspect with removed: false for entities that don’t have one, but this is not always invoked automatically for all entity types or ingestion flows (<https://github.com/datahub-project/datahub/pull/6705|see code reference>).

Would you like more detail on how to enable or configure stateful ingestion for a specific connector, or on how the status aspect is handled in custom ingestion scenarios?

Sources:

3

Hi <@U08BYGU0CSU>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.