Python >= 3.8
Thanks to the walrus operator (:=) the solution is quite short. We read bytes objects from the file and assign them to the variable byte
with open("myfile", "rb") as f:
while (byte := f.read(1)):
# Do stuff with byte.
Python >= 3
In older Python 3 versions, we get have to use a slightly more verbose way:
with open("myfile", "rb") as f:
byte = f.read(1)
while byte != b"":
# Do stuff with byte.
byte = f.read(1)
Or as benhoyt says, skip the not equal and take advantage of the fact that b"" evaluates to false. This makes the code compatible between 2.6 and 3.x without any changes. It would also save you from changing the condition if you go from byte mode to text or the reverse.
with open("myfile", "rb") as f:
byte = f.read(1)
while byte:
# Do stuff with byte.
byte = f.read(1)
Python >= 2.5
In Python 2, it's a bit different. Here we don't get bytes objects, but raw characters:
with open("myfile", "rb") as f:
byte = f.read(1)
while byte != "":
# Do stuff with byte.
byte = f.read(1)
Note that the with statement is not available in versions of Python below 2.5. To use it in v 2.5 you'll need to import it:
from __future__ import with_statement
In 2.6 this is not needed.
Python 2.4 and Earlier
f = open("myfile", "rb")
try:
byte = f.read(1)
while byte != "":
# Do stuff with byte.
byte = f.read(1)
finally:
f.close()
Python >= 3.8
Thanks to the walrus operator (:=) the solution is quite short. We read bytes objects from the file and assign them to the variable byte
with open("myfile", "rb") as f:
while (byte := f.read(1)):
# Do stuff with byte.
Python >= 3
In older Python 3 versions, we get have to use a slightly more verbose way:
with open("myfile", "rb") as f:
byte = f.read(1)
while byte != b"":
# Do stuff with byte.
byte = f.read(1)
Or as benhoyt says, skip the not equal and take advantage of the fact that b"" evaluates to false. This makes the code compatible between 2.6 and 3.x without any changes. It would also save you from changing the condition if you go from byte mode to text or the reverse.
with open("myfile", "rb") as f:
byte = f.read(1)
while byte:
# Do stuff with byte.
byte = f.read(1)
Python >= 2.5
In Python 2, it's a bit different. Here we don't get bytes objects, but raw characters:
with open("myfile", "rb") as f:
byte = f.read(1)
while byte != "":
# Do stuff with byte.
byte = f.read(1)
Note that the with statement is not available in versions of Python below 2.5. To use it in v 2.5 you'll need to import it:
from __future__ import with_statement
In 2.6 this is not needed.
Python 2.4 and Earlier
f = open("myfile", "rb")
try:
byte = f.read(1)
while byte != "":
# Do stuff with byte.
byte = f.read(1)
finally:
f.close()
This generator yields bytes from a file, reading the file in chunks:
def bytes_from_file(filename, chunksize=8192):
with open(filename, "rb") as f:
while True:
chunk = f.read(chunksize)
if chunk:
for b in chunk:
yield b
else:
break
# example:
for b in bytes_from_file('filename'):
do_stuff_with(b)
See the Python documentation for information on iterators and generators.
Python, how to read bytes from file and save it? - Stack Overflow
Fastest way to read bytes as a scalar?
How to Read Binary File in Python?
How to read a file byte by byte in Python and how to print a bytelist as a binary? - Stack Overflow
Videos
Here's how to do it with the basic file operations in Python. This opens one file, reads the data into memory, then opens the second file and writes it out.
in_file = open("in-file", "rb") # opening for [r]eading as [b]inary
data = in_file.read() # if you only wanted to read 512 bytes, do .read(512)
in_file.close()
out_file = open("out-file", "wb") # open for [w]riting as [b]inary
out_file.write(data)
out_file.close()
We can do this more succinctly by using the with keyboard to handle closing the file.
with open("in-file", "rb") as in_file, open("out-file", "wb") as out_file:
out_file.write(in_file.read())
If you don't want to store the entire file in memory, you can transfer it in pieces.
chunk_size = 4096 # 4 KiB
with open("in-file", "rb") as in_file, open("out-file", "wb") as out_file:
while True:
chunk = in_file.read(chunk_size)
if chunk == b"":
break # end of file
out_file.write(chunk)
In my examples I use the 'b' flag ('wb', 'rb') when opening the files because you said you wanted to read bytes. The 'b' flag tells Python not to interpret end-of-line characters which can differ between operating systems. If you are reading text, then omit the 'b' and use 'w' and 'r' respectively.
This reads the entire file in one chunk using the "simplest" Python code. The problem with this approach is that you could run out memory when reading a large file:
ifile = open(input_filename,'rb')
ofile = open(output_filename, 'wb')
ofile.write(ifile.read())
ofile.close()
ifile.close()
This example is refined to read 1MB chunks to ensure it works for files of any size without running out of memory:
ifile = open(input_filename,'rb')
ofile = open(output_filename, 'wb')
data = ifile.read(1024*1024)
while data:
ofile.write(data)
data = ifile.read(1024*1024)
ofile.close()
ifile.close()
This example is the same as above but leverages using with to create a context. The advantage of this approach is that the file is automatically closed when exiting the context:
with open(input_filename,'rb') as ifile:
with open(output_filename, 'wb') as ofile:
data = ifile.read(1024*1024)
while data:
ofile.write(data)
data = ifile.read(1024*1024)
See the following:
- Python open(): http://docs.python.org/library/functions.html#open
- Python read(): http://docs.python.org/library/stdtypes.html#file.read
- Python write(): http://docs.python.org/library/stdtypes.html#file.write
- Dive into Python File Object Tutorial: http://diveintopython.net/file_handling/file_objects.html
I'm reading in binary data from a file with:
data = np.fromfile(file,dtype='>b') data > array([ 5, 0, 0, ..., 2, 0, 99], dtype=int8)
This returns an array of int8. I'd like to read in, for example, the first 4 bytes and interpret these as an int32. I seem to have two options:
-
np.frombuffer(data[0:4],dtype=np.int32)[0]index the only element of a length-1 array. -
data[0:4].view(dtype=np.int32)[0]index the only element of a length-1 array.
Is there a 3rd option that directly reads this in as an int? It's in a loop and I don't want the overhead of constructing an array each time, followed by indexing the 0th element of the array. This seems unnecessary. Can't I just create an np.int32 from the first 4 bytes?
Update: here are the runtimes for the various options after running through some of the data:
| Method | Time |
|---|---|
int.from_bytes(data[idx:idx+4],byteorder='little',signed=False)
| 4.85s |
struct.unpack('<i',data[idx:idx+4])[0]
| 12.5s |
np.frombuffer(data[0:4],count=1,dtype=np.int32)[0]
| 31.3s |
np.frombuffer(data[0:4],dtype=np.int32)[0](leaving out count=1)
| 21.8s |
data[idx:idx+4].view(dtype=np.int32)[0]
| 16.6s |
The first one is the clear winner.
To read one byte:
file.read(1)
8 bits is one byte.
To answer the second part of your question, to convert to binary you can use a format string and the ord function:
>>> byte = 'a'
>>> '{0:08b}'.format(ord(byte))
'01100001'
Note that the format pads with the right number of leading zeros, which seems to be your requirement. This method needs Python 2.6 or later.