Setting Up a Custom Integration Between Informatica IICS and DataHub

ingestion
1

Original Slack Thread

Hi. Has anyone worked on Informatica IICS connect to Datahub

2

Hey there! :wave:

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!

3

It could be possible that there isn’t a direct, out-of-the-box integration between Informatica IICS (Informatica Intelligent Cloud Services) and DataHub. However, there might be ways to achieve this integration using APIs or custom connectors.

Based on a <Slack conversation>, it seems that while there isn’t a specific group known to have integrated Informatica Cloud Data Quality with DataHub, DataHub does have an event-oriented specification for external systems to report data quality rules and run results. Implementations for tools like Great Expectations and dbt tests are available in the open-source repository, which might serve as a reference for building a custom integration with Informatica IICS.

If you’re considering building this integration, you might want to explore using the DataHub API to report metadata and data quality results from Informatica IICS. It might also be helpful to look into how other integrations, such as those with Spark or Airflow, are documented and implemented.

Would you like more detailed guidance on how to approach building a custom integration between Informatica IICS and DataHub?

Sources:

  • <Slack conversation> :+1: 0 button :-1: 0 button :light_bulb: Hint: Mention @runllm in the thread for followups.
4

yes please

5

Yes please @<@U06TM7M40RK>

6

To integrate Informatica IICS with DataHub, you might consider building a custom integration using the DataHub API. Here’s a step-by-step approach you could take:

  1. Understand DataHub’s API: Familiarize yourself with the <https://datahubproject.io/docs/0.14.0/managed-datahub/datahub-api/entity-events-api|DataHub API> to understand how to send metadata and data quality information to DataHub. The API allows for real-time integration of changes happening on the DataHub Metadata Graph into a broader event-based architecture.

  2. Identify Metadata to Ingest: Determine what metadata from Informatica IICS you want to ingest into DataHub. This could include data lineage, data quality metrics, or other relevant metadata.

  3. Develop a Custom Connector:

    • Use Informatica IICS’s capabilities to extract the necessary metadata. This might involve using Informatica’s APIs or other export functionalities.
    • Develop a script or application that transforms this metadata into a format compatible with DataHub’s API. This might involve converting the metadata into JSON and structuring it according to DataHub’s requirements.

  4. Use DataHub’s Ingestion Framework:

    • You can use DataHub’s ingestion framework to create a custom ingestion source. Refer to the <https://datahubproject.io/docs/how/add-custom-ingestion-source/|documentation on adding a custom ingestion source> for guidance.
    • Implement the logic to send the transformed metadata to DataHub using the API. This might involve setting up a scheduled job or a real-time event listener that triggers the ingestion process.

  5. Testing and Validation:

    • Test the integration thoroughly to ensure that the metadata is correctly ingested into DataHub.
    • Validate that the data quality metrics and other metadata are accurately represented in DataHub.

  6. Documentation and Maintenance:

    • Document the integration process and any custom scripts or applications you develop.
    • Plan for ongoing maintenance to accommodate any changes in Informatica IICS or DataHub’s APIs.

This approach leverages DataHub’s extensibility and API capabilities to create a custom integration with Informatica IICS. If you encounter specific challenges or need further assistance, consider reaching out to the DataHub community for support.

Sources:

7

Out of curiosity: for what reason would you like to set up a connection between those two?

8

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><