S3 bucket data sync across different region and account

Overview

Transfer data from one AWS account to anohter AWS account. The S3 bucket located at different region and account. We have data synced operation at the destionation AWS account.

eg.

• source AWS accountID:1111222233334444
• source bucketName: your-source-bucket-name
• source region: us-east-1
• destionation AWS accountID:5555666677778888
• destion bucketName: your-destionation-bucket-name
• destionation region: us-east-2

Required permissions

Before start the data transfer operations, make sure that your source and destination AWS accounts have the right permissions to complete a cross-account transfer between S3 buckets.

Step 1: In your destionation account, create an IAM role for DataSync

Create the IAM role

Create an IAM role with DataSync as the trusted entity.

To create the IAM role

1. Log in to the AWS Management Console with your source account.
2. Open the IAM console at https://console.aws.amazon.com/iam/.
3. In the left navigation pane, under Access management, choose Roles, and then choose Create role.
4. On the Select trusted entity page, for Trusted entity type, choose AWS service.
5. For Use case, choose DataSync in the dropdown list and select DataSync. Choose Next.
6. On the Add permissions page, choose Next.
7. Give your role a name like datasync-config-role and choose Create role.

Attach a custom policy to the IAM role

To attach a custom policy to your IAM role

1. On the Roles page of the IAM console, search for the role that you just created and choose its name.
2. On the role's details page, choose the Permissions tab. Choose Add permissions then Create inline policy.
3. Choose the JSON tab and do the following:

   1. Paste the following JSON into the policy editor:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Action": [
              "s3:GetBucketLocation",
              "s3:ListBucket",
              "s3:ListBucketMultipartUploads"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::your-source-bucket-name"
          },
          {
            "Action": [
              "s3:AbortMultipartUpload",
              "s3:GetObject",
              "s3:ListMultipartUploadParts",
              "s3:GetObjectTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::your-source-bucket-name/*"
          },
         {
            "Action": [
              "s3:AbortMultipartUpload",
              "s3:DeleteObject",
              "s3:GetObject",
              "s3:ListMultipartUploadParts",
              "s3:PutObjectTagging",
              "s3:GetObjectTagging",
              "s3:PutObject"
                  ]
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::your-destionation-bucket-name/*"
          }
        ]
      }
      

   2. update the role permission

Step 2: In your destination account, disable ACLs for your S3 bucket

It's important that all the data that you transfer to the S3 bucket belongs to your destination account. To ensure that this account owns the data, disable the bucket's access control lists (ACLs).

To disable ACLs for an S3 bucket

1. In the AWS Management Console, switch over to your destination account.
2. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
3. In the left navigation pane, choose Buckets.
4. In the Buckets list, choose the S3 bucket that you're transferring data to.
5. On the bucket's detail page, choose the Permissions tab.
6. Under Object Ownership, choose Edit.
7. If it isn't already selected, choose the ACLs disabled (recommended) option.
8. Choose Save changes.

Step 3: In your source account, update your S3 bucket policy

In your destination account, modify the destination S3 bucket policy to include the DataSync IAM role that you created in your source account.

The updated bucket policy (provided to you in the following instructions) includes two principals:

• The first principal specifies the DataSync IAM role that you created in your source account. This role allows DataSync to write to the S3 bucket in your destination account.
• The second principal specifies the IAM role with the required user permissions for working with DataSync in your source account. You need this principal to create the DataSync destination location.

To update the destination S3 bucket policy

1. While still logged in to the S3 console with your destination account, choose the S3 bucket that you're transferring data to.

2. On the bucket's detail page, choose the Permissions tab.

3. Under Bucket policy, choose Edit and do the following to modify your S3 bucket policy:

   1. Update what's in the editor to include the following policy statements:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::5555666677778888:role/datasync-config-role"
           ]
         },
         "Action": [
           "s3:GetBucketLocation",
           "s3:ListBucket",
           "s3:ListBucketMultipartUploads"
         ],
         "Resource": [
           "arn:aws:s3:::your-source-bucket-name"
         ]
       },
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::5555666677778888:role/datasync-config-role"
           ]
         },
         "Action": [
           "s3:AbortMultipartUpload",
           "s3:GetObject",
           "s3:ListMultipartUploadParts",
           "s3:GetObjectTagging"
         ],
         "Resource": [
           "arn:aws:s3:::your-source-bucket-name/*"
         ]
       }
     ]
   }
   

Step 4: In your destionation account, create a DataSync source location

In your source account, you need to create a DataSync location for the S3 bucket in your destination account.

The DataSync console won't let you create locations for storage resources in another AWS account. However, you can do this by using AWS CloudShell, a browser-based, pre-authenticated shell that you launch directly from the console. CloudShell allows you to run the AWS CLI commands for completing this tutorial without downloading or installing command line tools.

To create a DataSync destination location by using CloudShell

1. In the AWS Management Console, switch back to your source account.

2. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

3. Do one of the following to launch CloudShell:

   • Choose the CloudShell icon on the console navigation bar. It's located to the right of the search box.
   • Use the search box on the console navigation bar to search for CloudShell and then choose the CloudShell option.

4. vi input.template, and save following code

   {
     "Subdirectory": "",
     "S3BucketArn": "arn:aws:s3:::your-source-bucket-name",
     "S3StorageClass": "STANDARD",
     "S3Config": {
       "BucketAccessRoleArn": "arn:aws:iam::5555666677778888:role/datasync-config-role"
     }
   }
   

5. Copy the following command:

   aws datasync create-location-s3 --cli-input-json file://input.template --region us-east-1
   

6. Run the command in CloudShell.

   if the command returns a DataSync location ARN similar to this, you successfully created the location:

{
    "LocationArn": "arn:aws:datasync:us-east-1:5555666677778888:location/loc-02f576071008e65ef"
}

Step 5: In your destionation account, create and start your DataSync transfer task

Before you move your data, let's recap what you've done so far:

• In your source account, you allow the role to access the object in source bucket
• In your destionation account, you configured your S3 bucket so that DataSync can access the bucket and write data to it.
• In your destination account, you created the DataSync source and destination locations for your transfer.

To create and start the DataSync transfer task

1. While still using the DataSync console in your source account, expand Data transfer in the left navigation pane, then choose Tasks and Create task.

2. If the bucket in your source account is in a different Region than the bucket in your destionation account, choose the source bucket's Region in the top navigation pane.

3. On the Configure source location page, do the following:

   1. Select Choose an existing location.
   2. (For transfers across Regions) In the Region dropdown, choose the Region where the source bucket resides.
   3. For Existing locations, choose the source location for the S3 bucket that you're transferring data from, then choose Next.

4. On the Configure destination location page, do the following:

   1. Select Choose an existing location.
   2. For Existing locations, choose the destination location for the S3 bucket that you're transferring data to, then choose Next.

5. On the Configure settings page, give the task a name. As needed, configure additional settings, such as specifying an Amazon CloudWatch log group. Choose Next.

6. On the Review page, review your settings and choose Create task.

7. On the task's details page, choose Start, and then choose one of the following:

   • To run the task without modification, choose Start with defaults.
   • To modify the task before running it, choose Start with overriding options.

When your task finishes, check the S3 bucket in your destination account. You should see the data that moved from your source account bucket.