yield is the keyword in python used for generator expressions. That means that the next time the function is called (or iterated on), the execution will start back up at the exact point it left off last time you called it. The two functions behave identically; the only difference is that the first one uses a tiny bit more call stack space than the second. However, the first one is far more reusable, so from a program design standpoint, the first one is actually better.
EDIT: Also, one other difference is that the first one will stop reading once all the data has been read, the way it should, but the second one will only stop once either f.read() or process_data() throws an exception. In order to have the second one work properly, you need to modify it like so:
f = open(file, 'rb')
while True:
piece = f.read(1024)
if not piece:
break
process_data(piece)
f.close()
Answer from AJMansfield on Stack Overflowyield is the keyword in python used for generator expressions. That means that the next time the function is called (or iterated on), the execution will start back up at the exact point it left off last time you called it. The two functions behave identically; the only difference is that the first one uses a tiny bit more call stack space than the second. However, the first one is far more reusable, so from a program design standpoint, the first one is actually better.
EDIT: Also, one other difference is that the first one will stop reading once all the data has been read, the way it should, but the second one will only stop once either f.read() or process_data() throws an exception. In order to have the second one work properly, you need to modify it like so:
f = open(file, 'rb')
while True:
piece = f.read(1024)
if not piece:
break
process_data(piece)
f.close()
starting from python 3.8 you might also use an assignment expression (the walrus-operator):
with open('file.name', 'rb') as file:
while chunk := file.read(1024):
process_data(chunk)
the last chunk may be smaller than CHUNK_SIZE.
as read() will return b"" when the file has been read the while loop will terminate.
I want to read 65 bytes from a binary file with a loop and write it as hex.
So I open the file;
from binascii import hexlify
L = [ ]
with open("btc.pdf", "rb") as bf:
L.append("c951" + "41" + hexlify(bf.read(65)) + "41" + hexlify(bf.read(65)) + "41" + hexlify(bf.read(65)))The problem is the file-size % 65 != 0. Furthermore, if I loop the above 1000 times the last 60 list items are all the same and consist of 40 bytes (from memory, I haven't got the code in front of me).
How is it done so a binary file is read in chunks and then stops when there's no more to read?
I'm working on a project that sends communication via a satellite. The communication sizes are limited to 430bytes.
What I'd like to do is take an image and split it up into ~430 byte chunks and send them to be reconstructed on the other side.
I was looking at the bytearray but I don't see how to specify how to break it up into specific bytes. I've also tried open(filename, 'rb') and I get what looks like byte chars but I dont know how to split them into specific chunks nor how to assemble them.
Has anyone seen something like this before?
edit: Here's the final code that works correctly. Thank you to everyone who helped make it happen
# SPLIT IMAGE APART
# Maximum chunk size that can be sent
CHUNK_SIZE = 430
# Location of source image
image_file = 'images/001.jpg'
# This file is for dev purposes. Each line is one piece of the message being sent individually
chunk_file = open('chunkfile.txt', 'wb+')
with open(image_file, 'rb') as infile:
while True:
# Read 430byte chunks of the image
chunk = infile.read(CHUNK_SIZE)
if not chunk: break
# Do what you want with each chunk (in dev, write line to file)
chunk_file.write(chunk)
chunk_file.close()
# STITCH IMAGE BACK TOGETHER
# Normally this will be in another location to stitch it back together
read_file = open('chunkfile.txt', 'rb')
# Create the jpg file
with open('images/stitched_together.jpg', 'wb') as image:
for f in read_file:
image.write(f)Since you are working with a file, the easiest is to simply read the file 430 bytes at a time.
CHUNK_SIZE = 430
f = open(image_file, 'rb')
chunk = f.read(CHUNK_SIZE)
while chunk: #loop until the chunk is empty (the file is exhausted)
send_chunk_to_space(chunk)
chunk = f.read(CHUNK_SIZE) #read the next chunk
f.close()
Did you try slicing and +
a = b'abcd'
print(type(a)) # <class 'bytes'>
print( a[3:] + a[:3] ) # b'dabc'
b = bytearray(a)
print(type(b)) # <class 'bytearray'>
print( b[3:] + b[:3] ) # bytearray(b'dabc')
Python >= 3.8
Thanks to the walrus operator (:=) the solution is quite short. We read bytes objects from the file and assign them to the variable byte
with open("myfile", "rb") as f:
while (byte := f.read(1)):
# Do stuff with byte.
Python >= 3
In older Python 3 versions, we get have to use a slightly more verbose way:
with open("myfile", "rb") as f:
byte = f.read(1)
while byte != b"":
# Do stuff with byte.
byte = f.read(1)
Or as benhoyt says, skip the not equal and take advantage of the fact that b"" evaluates to false. This makes the code compatible between 2.6 and 3.x without any changes. It would also save you from changing the condition if you go from byte mode to text or the reverse.
with open("myfile", "rb") as f:
byte = f.read(1)
while byte:
# Do stuff with byte.
byte = f.read(1)
Python >= 2.5
In Python 2, it's a bit different. Here we don't get bytes objects, but raw characters:
with open("myfile", "rb") as f:
byte = f.read(1)
while byte != "":
# Do stuff with byte.
byte = f.read(1)
Note that the with statement is not available in versions of Python below 2.5. To use it in v 2.5 you'll need to import it:
from __future__ import with_statement
In 2.6 this is not needed.
Python 2.4 and Earlier
f = open("myfile", "rb")
try:
byte = f.read(1)
while byte != "":
# Do stuff with byte.
byte = f.read(1)
finally:
f.close()
This generator yields bytes from a file, reading the file in chunks:
def bytes_from_file(filename, chunksize=8192):
with open(filename, "rb") as f:
while True:
chunk = f.read(chunksize)
if chunk:
for b in chunk:
yield b
else:
break
# example:
for b in bytes_from_file('filename'):
do_stuff_with(b)
See the Python documentation for information on iterators and generators.
You've got a few different options. Your main problem is that, with the small size of your chunks (6 bytes), there's a lot of overhead spent in fetching the chunk and garbage collecting.
There's two main ways to get around that:
Load the entire file into memory, THEN separate it into chunks. This is the fastest method, but the larger your file to more likely it is you will start running into MemoryErrors.
Load one chunk at a time into memory, process it, then move on to the next chunk. This is no faster overall, but saves time up front since you don't need to wait for the entire file to be loaded to start processing.
Experiment with combinations of 1. and 2. (buffering the file in large chunks and separating it into smaller chunks, loading the file in multiples of your chunk size, etc). This is left as an exercise for the viewer, as it will take a large amount of experimentation to reach code that will work quickly and correctly.
Some code, with time comparisons:
import timeit
def read_original(filename):
with open(filename, "rb") as infile:
data_arr = []
while True:
data = infile.read(6)
if not data:
break
data_arr.append(data)
return data_arr
# the bigger the file, the more likely this is to cause python to crash
def read_better(filename):
with open(filename, "rb") as infile:
# read everything into memory at once
data = infile.read()
# separate string into 6-byte chunks
data_arr = [data[i:i+6] for i in range(0, len(data), 6)]
return data_arr
# no faster than the original, but allows you to work on each piece without loading the whole into memory
def read_iter(filename):
with open(filename, "rb") as infile:
data = infile.read(6)
while data:
yield data
data = infile.read(6)
def main():
# 93.8688215 s
tm = timeit.timeit(stmt="read_original('test/oraociei12.dll')", setup="from __main__ import read_original", number=10)
print(tm)
# 85.69337399999999 s
tm = timeit.timeit(stmt="read_better('test/oraociei12.dll')", setup="from __main__ import read_better", number=10)
print(tm)
# 103.0508528 s
tm = timeit.timeit(stmt="[x for x in read_iter('test/oraociei12.dll')]", setup="from __main__ import read_iter", number=10)
print(tm)
if __name__ == '__main__':
main()
This way is much faster.
import sys
from functools import partial
SIX = 6
MULTIPLIER = 30000
SIX_COUNT = SIX * MULTIPLIER
def do(data):
for chunk in iter(partial(data.read, SIX_COUNT), b""):
six_list = [ chunk[i:i+SIX] for i in range(0, len(chunk), SIX) ]
if __name__ == "__main__":
args = sys.argv[1:]
for arg in args:
with open(arg,'rb') as data:
do(data)