One way to see the contents would be:
for my_bucket_object in my_bucket.objects.all():
print(my_bucket_object)
Top answer 1 of 16
407
One way to see the contents would be:
for my_bucket_object in my_bucket.objects.all():
print(my_bucket_object)
2 of 16
169
This is similar to an 'ls' but it does not take into account the prefix folder convention and will list the objects in the bucket. It's left up to the reader to filter out prefixes which are part of the Key name.
In Python 2:
from boto.s3.connection import S3Connection
conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
print(obj.key)
In Python 3:
from boto3 import client
conn = client('s3') # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
print(key['Key'])
DEV Community
dev.to › aws-builders › how-to-list-contents-of-s3-bucket-using-boto3-python-47mm
How to List Contents of s3 Bucket Using Boto3 Python? - DEV Community
October 12, 2021 - It'll list the files of that specific type from the Bucket and including all subdirectories. Use the below snippet to list specific file types from an S3 bucket. ... import boto3 session = boto3.Session( aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>') s3 = session.resource('s3') my_bucket = s3.Bucket('stackvidhya') for obj in my_bucket.objects.all(): if obj.key.endswith('txt'): print(obj.key)
listing the top level contents of a s3 bucket with Prefix and Delimiter
Apologies for what sounds like a very basic question. In this example from the s3 docs is there a way to list the continents? I was hoping this might work, but it doesn't seem to: import boto3 ... More on github.com
17
June 17, 2015Most efficient (cost/speed) to check if multiple files are in a S3 "folder" with Python/Boto3
We then have a list of file_ids as a list, between 1-100 in size (["file_id1", "file_id2", "file_id3"...]). Currently I loop through each file_id and do a try/except to request the file. If it fails to load, it doesn't exist and it's added to a separate process for creation. This feels costly/inefficient. What would you recommend as the quickest way of finding the file_ids that are NOT currently in the bucket/folder using Python/Boto3... More on repost.aws
2
0
June 9, 2023Using boto3 to list only all key names of files in a bucket - best methods when the bucket is ridiculously big?
Take a look at S3 Inventory. It's not realtime, but it's a better option than doing a full key enumeration when dealing with large buckets. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html More on reddit.com
4
2
June 20, 2021python - Retrieving subfolders names in S3 bucket from Boto3 - Stack Overflow
Now, the bucket contains folder first-level, which itself contains several sub-folders named with a timestamp, for instance 1456753904534. I need to know the name of these sub-folders for another job I'm doing and I wonder whether I could have boto3 retrieve those for me. ... which gives a dictionary, whose key 'Contents' gives me all the third-level files instead of the second-level timestamp directories, in fact I get a list ... More on stackoverflow.com
19:45
Boto3 Tutorial - List S3 Buckets - YouTube
04:34
Connect to AWS S3 and List Buckets with Python | Boto3 Tutorial ...
03:33
How to list files in S3 using Python | AWS S3 Python Boto3 | Step ...
2 - List All Objects In All S3 Buckets - Boto3 Basics
07:33
AWS S3 bucket | download all files using python programming language ...
AWS
docs.aws.amazon.com › boto3 › latest › reference › services › s3.html
S3 - Boto3 1.43.58 documentation
Resources are available in boto3 via the resource method. For more detailed instructions and examples on the usage of resources, see the resources user guide. ... The following example shows how to use an Amazon S3 bucket resource to list the objects in the bucket.
Habil
habil.dev › how-to-list-aws-s3-directory-contents-using-python-and-boto3
How to List AWS S3 Directory Contents Using Python and Boto3
May 21, 2026 - import boto3 import csv def list_bucket(): # Configuration bucket = "your-bucket-name" folder = "path/to/folder" # Initialize S3 client s3 = boto3.resource("s3", aws_access_key_id="YOUR_ACCESS_KEY", aws_secret_access_key="YOUR_SECRET_KEY" ) # Get bucket reference s3_bucket = s3.Bucket(bucket) # List files and extract relative paths files_in_s3 = [ f.key.split(folder + "/")[1] for f in s3_bucket.objects.filter(Prefix=folder).all() ] # Write results to file with open('bucket-contents.txt', 'w', encoding='UTF8') as file: file.write(str(files_in_s3)) if name == 'main': list_bucket()
GitHub
github.com › boto › boto3 › issues › 134
listing the top level contents of a s3 bucket with Prefix and Delimiter · Issue #134 · boto/boto3
June 17, 2015 - import boto3 s3 = boto3.resource('s3') bucket = s3.Bucket('edsu-test-bucket') for o in bucket.objects.filter(Delimiter='/'): print(o.key) However, the equivalent code using boto2 does seem to work the way I expect: import boto s3 = boto.connect_s3() bucket = s3.get_bucket('edsu-test-bucket') for o in bucket.list(delimiter='/'): print(o.name) Reactions are currently unavailable ·
Author boto
Binary Guy
binaryguy.tech › aws › s3 › quickest-ways-to-list-files-in-s3-bucket
Quickest Ways to List Files in S3 Bucket - Binary Guy
November 24, 2024 - Let us see how we can use paginator. def list_s3_files_using_paginator(): """ This functions list all files in s3 using paginator. Paginator is useful when you have 1000s of files in S3.
Top answer 1 of 2
2
To check for the existence of multiple files in an S3 "folder" using Python and Boto3, the most efficient method would be to take advantage of S3's prefix and delimiter options in the list_objects_v2 operation. However, you have rightly pointed out that there is a limitation with AWS, where the list_objects_v2 operation returns up to a maximum of 1000 keys (objects) at a time.
Here are two possible solutions:
Pagination: AWS SDKs return paginated results when the response is too large to handle. AWS uses a Pagination interface, which can be used to retrieve all the objects in a bucket, regardless of the number. You can use Boto3's built-in paginators to work around the limitation. This approach would involve listing all objects in the S3 bucket and comparing them with your list of file_ids. However, please note that if you have millions of files in your S3 bucket, this operation could be time-consuming and potentially cost more, as you are charged for LIST requests.
Parallelized checking: Instead of checking the files one by one, you could use a parallelized approach where you check the existence of multiple files concurrently. This could be done by using multi-threading or multi-processing in Python. However, this could potentially lead to higher costs due to increased GET requests, and you would need to handle exceptions for rate limiting (too many requests in a short period).
An important point to consider is that the cost-efficiency and speed of these methods can depend on the specific use case and circumstances, such as the size of the S3 bucket and the number of file_ids you are checking.
As an alternative to using Python and Boto3, you might consider using AWS Glue. AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. AWS Glue consists of a central data catalog, ETL (extract, transform, and load) engine, and flexible scheduler. AWS Glue can catalog your data, clean it, enrich it, and move it reliably between various data stores. Glue also provides various interfaces for different types of users including data scientists, data analysts, and ETL developers.
2 of 2
0
If you have already the file names and also the file prefix, you can build the object key instead of listing and verifying the bucket content. Listing and checking if it exists will be an expensive operation, instead you can do a HEAD and check if the file exists, it will be better and less expensive than doing GET.
For example you can create a function like that:
```python
def s3_exists(s3_bucket, s3_key):
try:
boto3.client('s3').head_object(Bucket=s3_bucket,
Key=s3_key)
return True
except:
return False
```
And later loop through the list calling this function, you can later add parallelism to improve execution time.
Reddit
reddit.com › r/aws › using boto3 to list only all key names of files in a bucket - best methods when the bucket is ridiculously big?
r/aws on Reddit: Using boto3 to list only all key names of files in a bucket - best methods when the bucket is ridiculously big?
June 20, 2021 -
Till now, to check the names or last_modified details of all objects in a bucket, what I did was the following:
import boto3
s3r = boto3.resource('s3', aws_access_key_id=access_key_id, aws_secret_access_key=access_key_secret)
bucket_items = list(s3r.Bucket('BucketName').objects.all())
However, this seems to be a terrible idea for handling it when the bucket gets too big. Running this on a machine with 8GB RAM threw a memory error!
I assume I have to be doing something wrong if only fetching the file keys is this difficult. Is the objects.all() method actually saving details of the file's contents into memory when I call it?
Top answer 1 of 4
3
Take a look at S3 Inventory. It's not realtime, but it's a better option than doing a full key enumeration when dealing with large buckets. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html
2 of 4
2
Having a hard time finding the reference material for this specific call in the boto3 docs, though it sounds suspiciously like it's not going to paginate the response, which is probably what you want to do for large buckets.. Using something like list_objects_v2(), will paginate the response and you can dump it into a file before making the next list call with the continuation token to get the next set of objects.. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2
AWS
docs.aws.amazon.com › boto3 › latest › reference › services › s3 › client › list_objects.html
list_objects - Boto3 1.43.57 documentation
When using the URL encoding type, ... test_file(3).png will appear as test_file(3).png. Marker (string) – Marker is where you want Amazon S3 to start listing from. Amazon S3 starts listing after this specified key. Marker can be any key in the bucket....
Medium
medium.com › @geekycodes › how-to-list-and-print-the-content-of-a-s3-bucket-in-aws-using-python-9200dfd89d8b
How to List and Print the Content of a S3 Bucket in AWS using Python | by Ved Prakash | Medium
August 2, 2023 - import boto3 def list_folders_in_bucket(bucket_name): # Create a Boto3 client for S3 s3_client = boto3.client('s3') # Use the list_objects_v2 method to get the list of objects in the bucket response = s3_client.list_objects_v2(Bucket=bucket_name, ...
Top answer 1 of 16
186
Below piece of code returns ONLY the 'subfolders' in a 'folder' from s3 bucket.
import boto3
bucket = 'my-bucket'
#Make sure you provide / in the end
prefix = 'prefix-name-with-slash/'
client = boto3.client('s3')
result = client.list_objects(Bucket=bucket, Prefix=prefix, Delimiter='/')
for o in result.get('CommonPrefixes'):
print 'sub folder : ', o.get('Prefix')
For more details, you can refer to https://github.com/boto/boto3/issues/134
2 of 16
126
S3 is an object storage, it doesn't have real directory structure. The "/" is rather cosmetic. One reason that people want to have a directory structure, because they can maintain/prune/add a tree to the application. For S3, you treat such structure as sort of index or search tag.
To manipulate object in S3, you need boto3.client or boto3.resource, e.g. To list all object
import boto3
s3 = boto3.client("s3")
all_objects = s3.list_objects(Bucket = 'bucket-name')
http://boto3.readthedocs.org/en/latest/reference/services/s3.html#S3.Client.list_objects
In fact, if the s3 object name is stored using '/' separator. The more recent version of list_objects (list_objects_v2) allows you to limit the response to keys that begin with the specified prefix.
To limit the items to items under certain sub-folders:
import boto3
s3 = boto3.client("s3")
response = s3.list_objects_v2(
Bucket=BUCKET,
Prefix ='DIR1/DIR2',
MaxKeys=100 )
Documentation
Another option is using python os.path function to extract the folder prefix. Problem is that this will require listing objects from undesired directories.
import os
s3_key = 'first-level/1456753904534/part-00014'
filename = os.path.basename(s3_key)
foldername = os.path.dirname(s3_key)
# if you are not using conventional delimiter like '#'
s3_key = 'first-level#1456753904534#part-00014'
filename = s3_key.split("#")[-1]
A reminder about boto3 : boto3.resource is a nice high level API. There are pros and cons using boto3.client vs boto3.resource. If you develop internal shared library, using boto3.resource will give you a blackbox layer over the resources used.
Alexwlchan
alexwlchan.net › 2017 › listing-s3-keys
Listing keys in an S3 bucket with Python – alexwlchan
It’s been very useful to have a list of files (or rather, keys) in the S3 bucket – for example, to get an idea of how many files there are to process, or whether they follow a particular naming scheme. The AWS APIs (via boto3) do provide a way to get this information, but API calls are paginated and don’t expose key names directly.
Medium
aws.plainenglish.io › aws-lambda-code-to-list-folders-in-s3-bucket-cff7d9ec5726
AWS Lambda: Get a List of Folders in the S3 Bucket | by John Sener | AWS in Plain English
January 29, 2024 - In this article, we’ll discuss a simple Python code snippet that lists folders in an S3 bucket using the boto3 library. import boto3 import botocore def list_files_in_folder(bucket_name, folder_prefix): s3 = boto3.client('s3') try: response = s3.list_objects(Bucket=bucket_name, Prefix=folder_prefix) if 'Contents' in response: return [obj['Key'] for obj in response['Contents']] else: return [] except botocore.exceptions.ClientError as e: print(f"Error listing files in the folder: {e}") return []
Reddit
reddit.com › r/aws › how do you list the most recent subfolder on s3 using boto3?
r/aws on Reddit: How do you list the most recent subfolder on s3 using boto3?
May 15, 2021 -
Lets say that My_bucket are the bucket that are public and there are /Cupcake/Vanilla/ subfolders. how do you get the most recent created subfolder on boto3?
Top answer 1 of 5
38
S3 is a key-based store, the concept of folders and subfolders does not exist beyond the AWS GUI which itself is just splitting keys on '/' characters for the sake of user convenience. It would be somewhat roundabout, but you could use the boto3 api to list all keys starting with a prefix such as /Cupcake/ and manually sift through the modified/creation timestamps to find the newest unique entry
2 of 5
5
Here's a script I've written to retrieve the last modified object using boto3 Hopefully, this will be of some help. As u/msgh pointed out , s3 doesn't differentiate between files and folders and everything is a object which has key as one of it's properites. import boto3 BUCKET = 'genxinputfiles' s3 = boto3.resource('s3') my_bucket = s3.Bucket(BUCKET) unsorted = [] for file in my_bucket.objects.filter(): unsorted.append(file) unsorted.sort(key=lambda x: x.last_modified, reverse=True) LastModifiedObject = unsorted[0].key
AWS re:Post
repost.aws › questions › QUbKRFAQCbTIa_b98zVOrbvA › fetching-the-file-in-the-s3-bucket
Fetching the file in the S3 bucket | AWS re:Post
July 26, 2023 - My bucket contains 80 customers geospatial data each customer have one directory. In the one directory have more than 50 directories. I am trying to go into the each file using python and boto3 but...
Radish Logic
radishlogic.com › home › list files in an s3 bucket folder
List files in an S3 Bucket folder Archives - Radish Logic
If you want to list the files/objects inside a specific folder within an S3 bucket then you will need to use the list_objects_v2 method with the Prefix parameter in boto3.
Brainly
brainly.com › engineering › college › how do i get the list of files in an s3 folder using python?
[FREE] How do I get the list of files in an S3 folder using Python? - brainly.com
Import boto3: Start your Python script by importing the boto3 library: ... Create an S3 Client: You will create a client that allows you to interact with S3. You can do this using: ... List Files in an S3 Folder: You can now list files in a specific S3 folder (which is treated as a prefix). Here’s how: # Replace with your S3 bucket name and folder path bucket_name = 'your-bucket-name' folder_path = 'your-folder-path/' # Get list of objects in the folder objects = s3.list_objects_v2(Bucket=bucket_name, Prefix=folder_path) # Check if objects were returned if 'Contents' in objects: for obj in objects['Contents']: print(obj['Key']) # This will print the file names else: print('No files found.')
GeeksforGeeks
geeksforgeeks.org › devops › how-to-retrieve-the-sub-folders-names-in-s3-bucket-using-boto3
How To Retrieve the Sub Folders Names In S3 Bucket Using Boto3? - GeeksforGeeks
July 23, 2025 - Prefix: A prefix is a string that can filter objects within a bucket. They work similarly to directories in a filesystem, but S3 does not have real directories. Prefixes can provide additional structure to objects by their name. Delimiter: A delimiter is a character used to group objects into a hierarchy. In S3 the most common delimiter is the forward slash (/). When applied with prefixes, it imitates folder structures that group objects with common prefixes. Paginator: In boto3, a paginator is an object that lets you iterate over pages of results returned by an API call.