Get an entire bucket or more than one object from an AWS S3 bucket through Ansible

Question

As far as I know the Ansible S3 module, it can immediately get an object.

My question is if I want to load / get a whole bucket or more than one object from an S3 bucket right away. Is there a hack?

Answer 1

I was able to achieve this like this:

- name: get s3_bucket_items s3: mode=list bucket=MY_BUCKET prefix=MY_PREFIX/ register: s3_bucket_items - name: download s3_bucket_items s3: mode=get bucket=MY_BUCKET object={{ item }} dest=/tmp/ with_items: s3_bucket_items.s3_keys 

Notes:

• Your prefix should not have a slash.
• The value {{ item }} will have a prefix already.

Answer 2

Unlike the S3 module, there is currently no built-in way to recursively sync buckets to disk.

In theory, you could try to collect keys for download using

 - name: register keys for syncronization s3: mode: list bucket: hosts object: /data/* register: s3_bucket_items - name: sync s3 bucket to disk s3: mode=get bucket=hosts object={{ item }} dest=/etc/data/conf/ with_items: s3_bucket_items.s3_keys 

Although I often see this solution, it does not seem to work with current versions of ansible / boto due to an error with S3 subdirectories (see this error report for more information), as well as an indispensable S3 module that does not create subdirectories for keys. I believe that it is also possible that you will encounter some memory problems using this method when synchronizing very large buckets.

I would also like to add that you most likely do not want to use the credentials encoded in your books - I suggest you use IAM EC2 instance profiles , which are much safer and more convenient.

The solution that works for me will be as follows:

 - name: Sync directory from S3 to disk command: "s3cmd sync -q --no-preserve s3://hosts/{{ item }}/ /etc/data/conf/" with_items: - data 

Answer 3

As of Ansible 2.0, the S3 module includes a list action that allows you to list keys in a bucket.

If you are not ready to upgrade to Ansible 2.0, then another approach may be to use a tool like s3cmd and invoke it through the command module:

 - name: Get objects command: s3cmd ls s3://my-bucket/path/to/objects register: s3objects 

Answer 4

He will be able to:

 - name: Get s3 objects s3: bucket: your-s3-bucket prefix: your-object-directory-path mode: list register: s3_object_list - name: Create download directory file: path: "/your/destination/directory/path/{{ item | dirname }}" state: directory with_items: - "{{ s3_object_list.s3_keys }}" - name: Download s3 objects s3: bucket: your-s3-bucket object: "{{ item }}" mode: get dest: "/your/destination/directory/path/{{ item }}" with_items: - "{{ s3_object_list.s3_keys }}" 

Answer 5

The following code will display each file in each bucket <3> in the account. It starts as a role with group_vars / localhost / vault.yml containing AWS keys.

I still do not understand why the second, more direct method II does not work, but maybe someone can enlighten us.

 - name: List S3 Buckets aws_s3_bucket_facts: aws_access_key: "{{ aws_access_key_id }}" aws_secret_key: "{{ aws_secret_access_key }}" # region: "eu-west-2" register: s3_buckets #- debug: var=s3_buckets - name: Iterate buckets set_fact: app_item: "{{ item.name }}" with_items: "{{ s3_buckets.ansible_facts.buckets }}" register: app_result #- debug: var=app_result.results #.item.name <= does not work?? - name: Create Fact List set_fact: s3_bucketlist: "{{ app_result.results | map(attribute='item.name') | list }}" #- debug: var=s3_bucketlist - name: List S3 Bucket files - Method I - works local_action: module: aws_s3 bucket: "{{ item }}" aws_access_key: "{{ aws_access_key_id }}" aws_secret_key: "{{ aws_secret_access_key }}" mode: list with_items: - "{{ s3_bucketlist }}" register: s3_list_I #- debug: var=s3_list_I - name: List S3 Bucket files - Method II - does not work aws_s3: aws_access_key: "{{ aws_access_key_id }}" aws_secret_key: "{{ aws_secret_access_key }}" bucket: "{{ item }}" mode: list with_items: "{{ s3_bucketlist }}" register: s3_list_II 

Answer 6

Perhaps you could change your "with_items", then you should work

  - name: get list to download
     aws_s3:
       region: "{{region}}"
       bucket: "{{item}}"
       mode: list
     with_items: "{{s3_bucketlist}}"
     register: s3_bucket_items

but maybe fast:

  - name: Sync directory from S3 to disk
     command: "aws --region {{region}} s3 sync s3: // {{bucket}} / / tmp / test"