Configuring DataHub for Ingesting DynamoDB Tables from Specific AWS Accounts

ingestion
21

Here is my usecase. I have a role called datahub-ingestion which I use with a service account. I want to ingest dynamodb tables from multiple accounts. What is the best approach? <@U06TM7M40RK>

22

To ingest DynamoDB tables from multiple AWS accounts using a single IAM role (datahub-ingestion) associated with a service account, the best approach is to configure cross-account access. This involves setting up trust relationships between the dev-services account (where the datahub-ingestion role resides) and the other accounts (test-dataops and dev-admin), allowing the datahub-ingestion role to assume roles in the other accounts.

Steps to Configure Cross-Account Access

  1. Create IAM Policies and Roles in Each Account: Define the necessary permissions and create IAM roles in each AWS account.
  2. Set Up Trust Relationships: Configure the trust relationships to allow the datahub-ingestion role in the dev-services account to assume roles in the test-dataops and dev-admin accounts.
  3. Configure the Ingestion Recipe: Set up the ingestion recipe to use the datahub-ingestion role and assume roles in the other accounts.

Example Configuration

Step 1: Create IAM Policies and Roles in Each Account
Create IAM policies with the required permissions for each AWS account. For example, for DynamoDB ingestion, you might need permissions like dynamodb:ListTables, dynamodb:DescribeTable, etc.

Example IAM Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:ListTables",
                "dynamodb:DescribeTable",
                "dynamodb:Scan"
            ],
            "Resource": "*"
        }
    ]
}

Create IAM roles in the test-dataops and dev-admin accounts and attach the respective policies.

Step 2: Set Up Trust Relationships
Configure the trust relationships to allow the datahub-ingestion role in the dev-services account to assume roles in the test-dataops and dev-admin accounts.

Example Trust Policy for test-dataops and dev-admin Accounts:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<dev-services-account-id>:role/datahub-ingestion"  # Replace with the ARN of the datahub-ingestion role
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Step 3: Configure the Ingestion Recipe
In your ingestion recipe, configure the datahub-ingestion role to assume roles in the test-dataops and dev-admin accounts.

Example Ingestion Recipe for Multiple Accounts:

23

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><