Using AWS IAM Roles for RDS Ingestion in DataHub: A Secure and Best Practice Approach

ingestion
1

Original Slack Thread

<@U06TM7M40RK> for RDS recipe’s - can I use aws IAM role’s to ingest a RDS db instead of username and password?

2

Yes, you can use AWS IAM roles for authentication instead of using a username and password for RDS ingestion in DataHub. This approach is generally more secure and aligns with AWS best practices for managing access to resources.

According to the <Slack Slack discussion>, DataHub supports IAM authentication for AWS RDS sources, particularly for PostgreSQL. You can enable IAM authentication by setting the EBEAN_POSTGRES_USE_AWS_IAM_AUTH environment variable and using the jdbc:aws-wrapper prefix in your connection string. This setup allows you to authenticate using IAM roles without needing to specify a username and password directly in your ingestion recipe.

Here’s a brief overview of how you can configure this:

  1. Set Environment Variable: Ensure that the environment variable EBEAN_POSTGRES_USE_AWS_IAM_AUTH is set to true.

  2. Configure the Connection String: Use the jdbc:aws-wrapper prefix in your connection string. For example:

    datasource:
  url: "jdbc:aws-wrapper:postgresql://{host}:5432/datahub"

  3. Grant IAM Access: Ensure that your IAM user has the necessary permissions to connect to the RDS instance. You can grant IAM access to the user in RDS with the following SQL command:

    GRANT rds_iam TO your_iam_user;

This setup is specifically mentioned for PostgreSQL, but the AWS wrapper jar should work generically for AWS databases, including MySQL, as discussed in the <Slack conversation>.

Would you like more detailed steps or specific configurations for your setup? If so, please let me know!

Sources:

  • <Slack Slack discussion on IAM auth for PostgreSQL> :+1: 0 button :-1: 0 button :light_bulb: Hint: Mention @runllm in the thread for followups.
3

Hi <@U06E9FUCM6Z>, how did my answer work out for you? Need any more assistance? You can also tag <@U07FV3UHE4D> to let your fellow community members know you’re still looking for guidance.