AWS DynamoDB auto-exit in S3 with an error with "role / DataPipelineDefaultRole is invalid"

Question

By following the step-by-step instructions on this page, I’m trying to export the contents of one of my DynamoDB tables to an S3 bucket. I create the pipeline exactly as indicated, but it does not start. It looks like he has problems defining / starting an EC2 resource for export. When I access EMR through the AWS Console, I see entries like this:

Cluster: df-0..._@EmrClusterForBackup _2015-03-06T00:33:04Terminated with errorsEMR service role arn:aws:iam::...:role/DataPipelineDefaultRole is invalid 

Why am I receiving this message? Do I need to configure / configure something else to start the pipeline?

UPDATE: in IAM->Roles in the AWS console, I see this for DataPipelineDefaultResourceRole :

 { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "s3:List*", "s3:Put*", "s3:Get*", "s3:DeleteObject", "dynamodb:DescribeTable", "dynamodb:Scan", "dynamodb:Query", "dynamodb:GetItem", "dynamodb:BatchGetItem", "dynamodb:UpdateTable", "rds:DescribeDBInstances", "rds:DescribeDBSecurityGroups", "redshift:DescribeClusters", "redshift:DescribeClusterSecurityGroups", "cloudwatch:PutMetricData", "datapipeline:PollForTask", "datapipeline:ReportTaskProgress", "datapipeline:SetTaskStatus", "datapipeline:PollForTask", "datapipeline:ReportTaskRunnerHeartbeat" ], "Resource": ["*"] }] } 

And this is for DataPipelineDefaultRole :

 { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "s3:List*", "s3:Put*", "s3:Get*", "s3:DeleteObject", "dynamodb:DescribeTable", "dynamodb:Scan", "dynamodb:Query", "dynamodb:GetItem", "dynamodb:BatchGetItem", "dynamodb:UpdateTable", "ec2:DescribeInstances", "ec2:DescribeSecurityGroups", "ec2:RunInstances", "ec2:CreateTags", "ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances", "elasticmapreduce:*", "rds:DescribeDBInstances", "rds:DescribeDBSecurityGroups", "redshift:DescribeClusters", "redshift:DescribeClusterSecurityGroups", "sns:GetTopicAttributes", "sns:ListTopics", "sns:Publish", "sns:Subscribe", "sns:Unsubscribe", "iam:PassRole", "iam:ListRolePolicies", "iam:GetRole", "iam:GetRolePolicy", "iam:ListInstanceProfiles", "cloudwatch:*", "datapipeline:DescribeObjects", "datapipeline:EvaluateExpression" ], "Resource": ["*"] }] } 

Do I need to change them somehow?

Answer 1

The AWS forum has a similar question, and it seems to be related to a managed policy issue.

https://forums.aws.amazon.com/message.jspa?messageID=606756

In this matter, they recommend that you use specific built-in policies for access and trust policies to identify roles that change some permissions. Oddly enough, specific built-in policies can be found in

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-iam-roles.html

Answer 2

I encountered the same error.

In IAM, attach the AWSDataPipelineRole managed policy to the DataPipelineDefaultRole

I also had to update the trust relation to the following (ec2 is needed, which is not in the documentation):

 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "ec2.amazonaws.com", "elasticmapreduce.amazonaws.com", "datapipeline.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] } 

Answer 3

I had the same problem. Managed policies were correct in my case, but I had to update trust relationships for the DataPipelineDefaultRole and DataPipelineDefaultResourceRole roles using the Gonfva documentation associated with the previous one, since they are deprecated.