Go through the following documentation
http://boto3.readthedocs.io/en/latest/guide/migrations3.html
Creating the Connection
Boto 3 has both low-level clients and higher-level resources. For Amazon S3, the higher-level resources are the most similar to Boto 2.x's s3 module:
Boto 2.x
import boto
s3_connection = boto.connect_s3()
Boto 3
import boto3
s3 = boto3.resource('s3')
Creating a Bucket
Creating a bucket in Boto 2 and Boto 3 is very similar, except that in Boto 3 all action parameters must be passed via keyword arguments and a bucket configuration must be specified manually:
Boto 2.x
s3_connection.create_bucket('mybucket')
s3_connection.create_bucket('mybucket', location=Location.USWest)
Boto 3
s3.create_bucket(Bucket='mybucket')
s3.create_bucket(Bucket='mybucket', CreateBucketConfiguration={
'LocationConstraint': 'us-west-1'})
Storing Data
Storing data from a file, stream, or string is easy:
Boto 2.x
from boto.s3.key import Key
key = Key('hello.txt')
key.set_contents_from_file('/tmp/hello.txt')
Boto 3
s3.Object('mybucket', 'hello.txt').put(Body=open('/tmp/hello.txt', 'rb'))
Go through the following documentation
http://boto3.readthedocs.io/en/latest/guide/migrations3.html
Creating the Connection
Boto 3 has both low-level clients and higher-level resources. For Amazon S3, the higher-level resources are the most similar to Boto 2.x's s3 module:
Boto 2.x
import boto
s3_connection = boto.connect_s3()
Boto 3
import boto3
s3 = boto3.resource('s3')
Creating a Bucket
Creating a bucket in Boto 2 and Boto 3 is very similar, except that in Boto 3 all action parameters must be passed via keyword arguments and a bucket configuration must be specified manually:
Boto 2.x
s3_connection.create_bucket('mybucket')
s3_connection.create_bucket('mybucket', location=Location.USWest)
Boto 3
s3.create_bucket(Bucket='mybucket')
s3.create_bucket(Bucket='mybucket', CreateBucketConfiguration={
'LocationConstraint': 'us-west-1'})
Storing Data
Storing data from a file, stream, or string is easy:
Boto 2.x
from boto.s3.key import Key
key = Key('hello.txt')
key.set_contents_from_file('/tmp/hello.txt')
Boto 3
s3.Object('mybucket', 'hello.txt').put(Body=open('/tmp/hello.txt', 'rb'))
First, in boto3, if you setup security using "aws configure" , you don't need to declare that "sess" section (http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
# if you already done aws configure
import boto3
s3 = boto3.client("s3")
s3.create_bucket(Bucket="mybucket", ....)
Second, is the bad boto3 documentation that fail to link proper information. This is found under boto3 pdf, page 2181 (https://media.readthedocs.org/pdf/boto3/latest/boto3.pdf)
Email : The value in the Grantee object is the registered email address of an AWS account.
Grantee : The AWS user or group that you want to have access to transcoded files and playlists. To identify the user or group, you can specify the canonical user ID for an AWS account, an origin access identity for a CloudFront distribution, the registered email address of an AWS account, or a predefined Amazon S3 group
And the easier solution is just use policy setting (http://support.cloudcheckr.com/getting-started-with-cloudcheckr/preparing-your-aws-account/aggregate-cloudtrail/) . You can convert the whole stuff using put_bucket_policy(), skip the dire GrantWrite,GrantWriteACP
» pip install boto3
There is a customization that went into Boto3 recently which helps with this (among other things). It is currently exposed on the low-level S3 client, and can be used like this:
s3_client = boto3.client('s3')
open('hello.txt').write('Hello, world!')
# Upload the file to S3
s3_client.upload_file('hello.txt', 'MyBucket', 'hello-remote.txt')
# Download the file from S3
s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2.txt')
print(open('hello2.txt').read())
These functions will automatically handle reading/writing files as well as doing multipart uploads in parallel for large files.
Note that s3_client.download_file won't create a directory. It can be created as pathlib.Path('/path/to/file.txt').parent.mkdir(parents=True, exist_ok=True).
boto3 now has a nicer interface than the client:
resource = boto3.resource('s3')
my_bucket = resource.Bucket('MyBucket')
my_bucket.download_file(key, local_filename)
This by itself isn't tremendously better than the client in the accepted answer (although the docs say that it does a better job retrying uploads and downloads on failure) but considering that resources are generally more ergonomic (for example, the s3 bucket and object resources are nicer than the client methods) this does allow you to stay at the resource layer without having to drop down.
Resources generally can be created in the same way as clients, and they take all or most of the same arguments and just forward them to their internal clients.
» pip install mypy-boto3-s3