yield is the keyword in python used for generator expressions. That means that the next time the function is called (or iterated on), the execution will start back up at the exact point it left off last time you called it. The two functions behave identically; the only difference is that the first one uses a tiny bit more call stack space than the second. However, the first one is far more reusable, so from a program design standpoint, the first one is actually better.

EDIT: Also, one other difference is that the first one will stop reading once all the data has been read, the way it should, but the second one will only stop once either f.read() or process_data() throws an exception. In order to have the second one work properly, you need to modify it like so:

f = open(file, 'rb')
while True:
    piece = f.read(1024)  
    if not piece:
        break
    process_data(piece)
f.close()
Answer from AJMansfield on Stack Overflow
🌐
Python Morsels
pythonmorsels.com › reading-binary-files-in-python
Reading binary files in Python - Python Morsels
May 16, 2022 - But instead of reading our entire file into memory, we're now reading our file chunk-by-chunk. It's common to see an assignment expression used (via Python's walrus operator) when reading files chunk-by-chunk:
🌐
Reddit
reddit.com › r/learnpython › reading chunks of a binary file with a loop
r/learnpython on Reddit: Reading chunks of a binary file with a loop
March 21, 2015 -

I want to read 65 bytes from a binary file with a loop and write it as hex.

So I open the file;

from binascii import hexlify
L = [ ]
with open("btc.pdf", "rb") as bf:
    L.append("c951" + "41" + hexlify(bf.read(65)) + "41" + hexlify(bf.read(65)) + "41" + hexlify(bf.read(65)))

The problem is the file-size % 65 != 0. Furthermore, if I loop the above 1000 times the last 60 list items are all the same and consist of 40 bytes (from memory, I haven't got the code in front of me).

How is it done so a binary file is read in chunks and then stops when there's no more to read?

🌐
DaniWeb
daniweb.com › programming › software-development › threads › 515440 › reading-an-entire-binary-file
python - Reading an entire binary file [SOLVED] | DaniWeb
January 22, 2018 - If you must process in portions, keep your output file open once and append each processed chunk; repeatedly reopening with mode "wb" will overwrite earlier chunks and leave only the last block. For even lower overhead, consider readinto() with a preallocated bytearray or memoryview, or, for advanced use cases, memory mapping (mmap). References: open(), Buffered I/O read semantics, binascii.hexlify, shutil.copyfileobj, hashlib. ... with open("portrait1.dng", "rb") as binaryfile : with open("readfile.raw", "wb") as newFile: while True: chunk = binaryfile.read(4096) if not chunk: break newFile.write(binascii.hexlify(chunk))
🌐
Reddit
reddit.com › r/learnpython › how do you split a binary file into specific sized chunks?
r/learnpython on Reddit: How do you split a binary file into specific sized chunks?
September 7, 2016 -

I'm working on a project that sends communication via a satellite. The communication sizes are limited to 430bytes.

What I'd like to do is take an image and split it up into ~430 byte chunks and send them to be reconstructed on the other side.

I was looking at the bytearray but I don't see how to specify how to break it up into specific bytes. I've also tried open(filename, 'rb') and I get what looks like byte chars but I dont know how to split them into specific chunks nor how to assemble them.

Has anyone seen something like this before?

edit: Here's the final code that works correctly. Thank you to everyone who helped make it happen

# SPLIT IMAGE APART
# Maximum chunk size that can be sent
CHUNK_SIZE = 430

# Location of source image
image_file = 'images/001.jpg'

# This file is for dev purposes. Each line is one piece of the message being sent individually
chunk_file = open('chunkfile.txt', 'wb+')

with open(image_file, 'rb') as infile:
    while True:
        # Read 430byte chunks of the image
        chunk = infile.read(CHUNK_SIZE)
        if not chunk: break

        # Do what you want with each chunk (in dev, write line to file)
        chunk_file.write(chunk)

chunk_file.close()


# STITCH IMAGE BACK TOGETHER
# Normally this will be in another location to stitch it back together
read_file = open('chunkfile.txt', 'rb')

# Create the jpg file
with open('images/stitched_together.jpg', 'wb') as image:
    for f in read_file:
        image.write(f)
🌐
Program Creek
programcreek.com › python
Python read chunks
def read_in_chunks( file_object: Union[BinaryIO, TextIO], block_size: int = 4096 ) -> Generator[Union[bytes, str], None, None]: """Return a generator which yields data in chunks.
🌐
GeeksforGeeks
geeksforgeeks.org › python › reading-binary-files-in-python
Reading Binary Files in Python - GeeksforGeeks
3 weeks ago - For large binary files, reading the entire file at once may consume a lot of memory. In such cases, it is better to read the file in smaller chunks using read(size).
🌐
Python Forum
python-forum.io › thread-8336.html
Read binary file
Hi, I am beginner and I need to read binary files. How can I read each forth(nth) chunk of 1024 bytes.
Find elsewhere
🌐
Java2s
java2s.com › example › python-book › read-file-chunk-by-chunk.html
Python - Read file chunk by chunk
Python · Statement · while Loops · Read file chunk by chunk · file = open('main.py', 'rb') while True: # from w w w .j av a 2 s .c o m chunk = file.read(10) # Read byte chunks: up to 10 bytes if not chunk: break print(chunk) You typically read binary data in blocks.
Top answer
1 of 13
493

Python >= 3.8

Thanks to the walrus operator (:=) the solution is quite short. We read bytes objects from the file and assign them to the variable byte

with open("myfile", "rb") as f:
    while (byte := f.read(1)):
        # Do stuff with byte.

Python >= 3

In older Python 3 versions, we get have to use a slightly more verbose way:

with open("myfile", "rb") as f:
    byte = f.read(1)
    while byte != b"":
        # Do stuff with byte.
        byte = f.read(1)

Or as benhoyt says, skip the not equal and take advantage of the fact that b"" evaluates to false. This makes the code compatible between 2.6 and 3.x without any changes. It would also save you from changing the condition if you go from byte mode to text or the reverse.

with open("myfile", "rb") as f:
    byte = f.read(1)
    while byte:
        # Do stuff with byte.
        byte = f.read(1)

Python >= 2.5

In Python 2, it's a bit different. Here we don't get bytes objects, but raw characters:

with open("myfile", "rb") as f:
    byte = f.read(1)
    while byte != "":
        # Do stuff with byte.
        byte = f.read(1)

Note that the with statement is not available in versions of Python below 2.5. To use it in v 2.5 you'll need to import it:

from __future__ import with_statement

In 2.6 this is not needed.

Python 2.4 and Earlier

f = open("myfile", "rb")
try:
    byte = f.read(1)
    while byte != "":
        # Do stuff with byte.
        byte = f.read(1)
finally:
    f.close()
2 of 13
194

This generator yields bytes from a file, reading the file in chunks:

def bytes_from_file(filename, chunksize=8192):
    with open(filename, "rb") as f:
        while True:
            chunk = f.read(chunksize)
            if chunk:
                for b in chunk:
                    yield b
            else:
                break

# example:
for b in bytes_from_file('filename'):
    do_stuff_with(b)

See the Python documentation for information on iterators and generators.

🌐
Stack Overflow
stackoverflow.com › questions › 71978290 › python-how-to-read-binary-file-by-chunks-and-specify-the-beginning-offset
Python how to read binary file by chunks and specify the beginning offset - Stack Overflow
Copydef read_chunks(infile, chunk_size): while True: chunk = infile.read(chunk_size) if chunk: yield chunk else: return · This works when I need to read the file by chunks; however, sometimes I need to read the file two bytes at a time, but start reading at the next offset, not the next chunk.
🌐
Python Guides
pythonguides.com › python-read-a-binary-file
How to Read a Binary File in Python
November 4, 2025 - file_path = "C:/Users/John/Documents/large_image_file.bin" # Define the chunk size (for example, 1024 bytes) chunk_size = 1024 with open(file_path, "rb") as binary_file: while chunk := binary_file.read(chunk_size): # Process each chunk (for example, print length) print(f"Read {len(chunk)} bytes") You can refer to the screenshot below to see the output. This approach is especially useful when you need to process binary data on the fly, such as streaming video frames or reading large log files. Python’s while loop combined with the assignment expression (:=) makes this method both elegant and efficient.
🌐
Finxter
blog.finxter.com › home › learn python blog › python read binary file
Python Read Binary File – Be on the Right Side of Change
April 10, 2022 - Line [1] assigns the size of the chunk to the variable chunk_size. Line [2] assigns the variable image_file to the file to be read in.
🌐
YouTube
youtube.com › python morsels
Reading binary files in Python - YouTube
How can you read binary files in Python? And how can you read very large binary files in small chunks?Read an article version of this video at https://pym.de...
Published   October 5, 2022
Views   5K
🌐
Quora
quora.com › How-can-I-read-a-large-file-in-Python-in-chunks-based-on-begin-and-end-string-values-Reading-them-by-byte-size-slices-it-in-the-middle-of-string-values-and-am-missing-data
How can I read a large file in Python in chunks based on begin and end string values? Reading them by byte size slices it in the middle o...
Answer (1 of 3): A file has no concept of a Python string (a file is just a sequence of bytes on a disk) and a Python string could be any sequence of characters - including the entire file if required. So the whole thing depends on your applications definition of the ‘begin and end string ...
🌐
IT trip
en.ittrip.xyz › python
How to Read Binary Files Using the Read Method in Python | IT trip
October 31, 2024 - Open a large binary file large_file.bin and read it in 1KB chunks, displaying the first 10 bytes of each chunk.
🌐
YouTube
youtube.com › watch
python read binary file in chunks
Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.
🌐
Accuweb
accuweb.cloud › home › how to read binary file in python?
How to Read Binary File in Python?
May 2, 2024 - chunk_size = 4096 with open("large.bin", "rb") as f: while chunk := f.read(chunk_size): process(chunk)Copy ... Binary files often contain structured fields like integers and floats. Python’s struct module converts raw bytes into Python values.
🌐
Appdividend
appdividend.com › read-binary-file-in-python
How to a Read Binary File in Python
September 6, 2025 - The argument “rb” means r stands for read-only and b stands for binary. There is one flaw in this approach. If the file is enormous, it may consume all available RAM. So, this approach is not efficient when the file size is extremely large. To resolve the memory exhaustion issue, we need to divide the file’s content into chunks by passing a chunk size to the f.read() method, and then read the file chunk by chunk, printing it if desired.
Top answer
1 of 2
2

You've got a few different options. Your main problem is that, with the small size of your chunks (6 bytes), there's a lot of overhead spent in fetching the chunk and garbage collecting.

There's two main ways to get around that:

  1. Load the entire file into memory, THEN separate it into chunks. This is the fastest method, but the larger your file to more likely it is you will start running into MemoryErrors.

  2. Load one chunk at a time into memory, process it, then move on to the next chunk. This is no faster overall, but saves time up front since you don't need to wait for the entire file to be loaded to start processing.

  3. Experiment with combinations of 1. and 2. (buffering the file in large chunks and separating it into smaller chunks, loading the file in multiples of your chunk size, etc). This is left as an exercise for the viewer, as it will take a large amount of experimentation to reach code that will work quickly and correctly.

Some code, with time comparisons:

import timeit


def read_original(filename):
    with open(filename, "rb") as infile:
        data_arr = []
        while True:
            data = infile.read(6)
            if not data:
                break
            data_arr.append(data)
    return data_arr


# the bigger the file, the more likely this is to cause python to crash
def read_better(filename):
    with open(filename, "rb") as infile:
        # read everything into memory at once
        data = infile.read()
        # separate string into 6-byte chunks
        data_arr = [data[i:i+6] for i in range(0, len(data), 6)]
    return data_arr

# no faster than the original, but allows you to work on each piece without loading the whole into memory
def read_iter(filename):
    with open(filename, "rb") as infile:
        data = infile.read(6)
        while data:
            yield data
            data = infile.read(6)


def main():
    # 93.8688215 s
    tm = timeit.timeit(stmt="read_original('test/oraociei12.dll')", setup="from __main__ import read_original", number=10)
    print(tm)
    # 85.69337399999999 s
    tm = timeit.timeit(stmt="read_better('test/oraociei12.dll')", setup="from __main__ import read_better", number=10)
    print(tm)
    # 103.0508528 s
    tm = timeit.timeit(stmt="[x for x in read_iter('test/oraociei12.dll')]", setup="from __main__ import read_iter", number=10)
    print(tm)

if __name__ == '__main__':
    main()
2 of 2
0

This way is much faster.

import sys
from functools import partial

SIX = 6
MULTIPLIER = 30000
SIX_COUNT = SIX * MULTIPLIER

def do(data):
    for chunk in iter(partial(data.read, SIX_COUNT), b""):
        six_list = [ chunk[i:i+SIX] for i in range(0, len(chunk), SIX) ]

if __name__ == "__main__": 
    args = sys.argv[1:]
    for arg in args:
        with open(arg,'rb') as data:
            do(data)