Go through the following documentation
http://boto3.readthedocs.io/en/latest/guide/migrations3.html
Creating the Connection
Boto 3 has both low-level clients and higher-level resources. For Amazon S3, the higher-level resources are the most similar to Boto 2.x's s3 module:
Boto 2.x
import boto
s3_connection = boto.connect_s3()
Boto 3
import boto3
s3 = boto3.resource('s3')
Creating a Bucket
Creating a bucket in Boto 2 and Boto 3 is very similar, except that in Boto 3 all action parameters must be passed via keyword arguments and a bucket configuration must be specified manually:
Boto 2.x
s3_connection.create_bucket('mybucket')
s3_connection.create_bucket('mybucket', location=Location.USWest)
Boto 3
s3.create_bucket(Bucket='mybucket')
s3.create_bucket(Bucket='mybucket', CreateBucketConfiguration={
'LocationConstraint': 'us-west-1'})
Storing Data
Storing data from a file, stream, or string is easy:
Boto 2.x
from boto.s3.key import Key
key = Key('hello.txt')
key.set_contents_from_file('/tmp/hello.txt')
Boto 3
s3.Object('mybucket', 'hello.txt').put(Body=open('/tmp/hello.txt', 'rb'))
Go through the following documentation
http://boto3.readthedocs.io/en/latest/guide/migrations3.html
Creating the Connection
Boto 3 has both low-level clients and higher-level resources. For Amazon S3, the higher-level resources are the most similar to Boto 2.x's s3 module:
Boto 2.x
import boto
s3_connection = boto.connect_s3()
Boto 3
import boto3
s3 = boto3.resource('s3')
Creating a Bucket
Creating a bucket in Boto 2 and Boto 3 is very similar, except that in Boto 3 all action parameters must be passed via keyword arguments and a bucket configuration must be specified manually:
Boto 2.x
s3_connection.create_bucket('mybucket')
s3_connection.create_bucket('mybucket', location=Location.USWest)
Boto 3
s3.create_bucket(Bucket='mybucket')
s3.create_bucket(Bucket='mybucket', CreateBucketConfiguration={
'LocationConstraint': 'us-west-1'})
Storing Data
Storing data from a file, stream, or string is easy:
Boto 2.x
from boto.s3.key import Key
key = Key('hello.txt')
key.set_contents_from_file('/tmp/hello.txt')
Boto 3
s3.Object('mybucket', 'hello.txt').put(Body=open('/tmp/hello.txt', 'rb'))
First, in boto3, if you setup security using "aws configure" , you don't need to declare that "sess" section (http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
# if you already done aws configure
import boto3
s3 = boto3.client("s3")
s3.create_bucket(Bucket="mybucket", ....)
Second, is the bad boto3 documentation that fail to link proper information. This is found under boto3 pdf, page 2181 (https://media.readthedocs.org/pdf/boto3/latest/boto3.pdf)
Email : The value in the Grantee object is the registered email address of an AWS account.
Grantee : The AWS user or group that you want to have access to transcoded files and playlists. To identify the user or group, you can specify the canonical user ID for an AWS account, an origin access identity for a CloudFront distribution, the registered email address of an AWS account, or a predefined Amazon S3 group
And the easier solution is just use policy setting (http://support.cloudcheckr.com/getting-started-with-cloudcheckr/preparing-your-aws-account/aggregate-cloudtrail/) . You can convert the whole stuff using put_bucket_policy(), skip the dire GrantWrite,GrantWriteACP
Below piece of code returns ONLY the 'subfolders' in a 'folder' from s3 bucket.
import boto3
bucket = 'my-bucket'
#Make sure you provide / in the end
prefix = 'prefix-name-with-slash/'
client = boto3.client('s3')
result = client.list_objects(Bucket=bucket, Prefix=prefix, Delimiter='/')
for o in result.get('CommonPrefixes'):
print 'sub folder : ', o.get('Prefix')
For more details, you can refer to https://github.com/boto/boto3/issues/134
S3 is an object storage, it doesn't have real directory structure. The "/" is rather cosmetic. One reason that people want to have a directory structure, because they can maintain/prune/add a tree to the application. For S3, you treat such structure as sort of index or search tag.
To manipulate object in S3, you need boto3.client or boto3.resource, e.g. To list all object
import boto3
s3 = boto3.client("s3")
all_objects = s3.list_objects(Bucket = 'bucket-name')
http://boto3.readthedocs.org/en/latest/reference/services/s3.html#S3.Client.list_objects
In fact, if the s3 object name is stored using '/' separator. The more recent version of list_objects (list_objects_v2) allows you to limit the response to keys that begin with the specified prefix.
To limit the items to items under certain sub-folders:
import boto3
s3 = boto3.client("s3")
response = s3.list_objects_v2(
Bucket=BUCKET,
Prefix ='DIR1/DIR2',
MaxKeys=100 )
Documentation
Another option is using python os.path function to extract the folder prefix. Problem is that this will require listing objects from undesired directories.
import os
s3_key = 'first-level/1456753904534/part-00014'
filename = os.path.basename(s3_key)
foldername = os.path.dirname(s3_key)
# if you are not using conventional delimiter like '#'
s3_key = 'first-level#1456753904534#part-00014'
filename = s3_key.split("#")[-1]
A reminder about boto3 : boto3.resource is a nice high level API. There are pros and cons using boto3.client vs boto3.resource. If you develop internal shared library, using boto3.resource will give you a blackbox layer over the resources used.
One way to see the contents would be:
for my_bucket_object in my_bucket.objects.all():
print(my_bucket_object)
This is similar to an 'ls' but it does not take into account the prefix folder convention and will list the objects in the bucket. It's left up to the reader to filter out prefixes which are part of the Key name.
In Python 2:
from boto.s3.connection import S3Connection
conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
print(obj.key)
In Python 3:
from boto3 import client
conn = client('s3') # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
print(key['Key'])