If they are all within the same line, that is there are no line breaks between "1." and "2." then you can iterate over the lines of the file like this:

for line in open("myfile.txt"):
    #do stuff

The line will be disposed of and overwritten at each iteration meaning you can handle large file sizes with ease. If they're not on the same line:

for line in open("myfile.txt"):
    if #regex to match start of new string
       parsed_line = line
    else:
       parsed_line += line

and the rest of your code.

Answer from wheaties on Stack Overflow
🌐
Reddit
reddit.com › r/learnpython › how do i read in a large file in chunks?
r/learnpython on Reddit: How do I read in a large file in chunks?
April 4, 2021 -

Hello,

I would like to know how to best approach the following task.

I have a large file which is a few million lines. I want to read each line and do something with it.

Considerations:

  • I don’t want the whole file loaded into memory at once, I want this loaded in chunks

  • Threading should be used (unless there’s a better option)

My initial thought process in pseudocode would be something like:

lines_at_once = 100000

for line in read(file, lines_at_once):
    do_something(line)
EOF when reading in chunks Dec 19, 2023
r/learnpython
2y ago
best way to process large files Oct 29, 2020
r/learnpython
5y ago
How to file.readline inside of a loop Aug 5, 2018
r/learnpython
8y ago
Issues reading large files? Mar 28, 2024
r/learnpython
2y ago
More results from reddit.com
🌐
Medium
fanchenbao.medium.com › chunk-read-a-large-file-in-python-4b887058215
Chunk Read A Large File in Python | by FanchenBao | Medium
June 11, 2022 - Essentially, they belong to two categories: pass a chunk size to the built-in read function or read the file line by line. The first method dictates how many characters (or bytes, if the read mode is binary) to read at one go, whereas the second method leverages the fact that Python always throws away the previous line when it reads the current one.
People also ask

What is the best way to read a very large file in Python
ANS: The most efficient methods involve reading the file in chunks rather than loading the entire content into memory. Techniques like using generators with yield, the iter() function, or the mmap module are highly recommended for memory-efficient processing of large files.
🌐
sqlpey.com
sqlpey.com › python › python-efficiently-read-large-files-in-chunks
Python: Efficiently Read Large Files in Chunks - sqlpey
When should I use the mmap module for file reading
ANS: The mmap module is beneficial for large files on 64-bit systems where you need random access to file content as if it were in memory. It maps the file directly, allowing operations similar to strings or byte arrays. However, use it cautiously on 32-bit systems due to potential address space limitations.
🌐
sqlpey.com
sqlpey.com › python › python-efficiently-read-large-files-in-chunks
Python: Efficiently Read Large Files in Chunks - sqlpey
How can I avoid memory errors when processing large files
ANS: Avoid methods that load the entire file at once, such as file.read() without arguments or file.readlines() without a size hint. Instead, opt for chunked reading using generators, iter(), or iterative calls to file.read(size).
🌐
sqlpey.com
sqlpey.com › python › python-efficiently-read-large-files-in-chunks
Python: Efficiently Read Large Files in Chunks - sqlpey
🌐
Stack Overflow
stackoverflow.com › questions › 519633 › lazy-method-for-reading-big-file-in-python
Lazy Method for Reading Big File in Python? - Stack Overflow
I have a very big file 4GB and when I try to read it my computer hangs. So I want to read it piece by piece and after processing each piece store the processed piece into another file and read next...
🌐
Quora
quora.com › How-can-I-read-a-large-file-in-Python-in-chunks-based-on-begin-and-end-string-values-Reading-them-by-byte-size-slices-it-in-the-middle-of-string-values-and-am-missing-data
How can I read a large file in Python in chunks based on begin and end string values? Reading them by byte size slices it in the middle o...
Answer (1 of 3): A file has no concept of a Python string (a file is just a sequence of bytes on a disk) and a Python string could be any sequence of characters - including the entire file if required. So the whole thing depends on your applications definition of the ‘begin and end string ...
🌐
Developer-service
developer-service.blog › handling-large-files-and-optimizing-file-operations-in-python
Handling Large Files and Optimizing File Operations in Python
September 2, 2025 - In this example, we specify a chunk size of 1 KB and read the file in chunks of that size. The while loop continues reading until there’s no more data to read (chunk is empty). This method is particularly useful for handling large binary files or when you need to work with specific byte ranges.
🌐
Reddit
reddit.com › r/learnpython › chunking in python---how to set the "chunk size" of read lines from file read with python open()?
r/learnpython on Reddit: Chunking in Python---How to set the "chunk size" of read lines from file read with Python open()?
September 15, 2016 -

I have a fairly large text file which I would like to run in chunks. In order to do this with the subprocess library, one would execute following shell command:

"cat hugefile.log"

with the code:

import subprocess
task = subprocess.Popen("cat hugefile.log", shell=True,  stdout=subprocess.PIPE)
data = task.stdout.read()

Using print(data) will spit out the entire contents of the file at once. How can I present the number of chunks, and then access the contents of this file by the chunk size (e.g. chunk = three lines at a time).

It must be something like:

chunksize = 1000   # break up hugefile.log into 1000 chunks

for chunk in data:
    print(chunk)

The equivalent question with Python open() of course uses the code

with open('hugefile.log', 'r') as f:
     read_data = f.read()

How would you read_data in chunks?

Find elsewhere
🌐
Medium
medium.com › @tubelwj › how-to-read-extremely-large-text-files-in-python-cddc7dbce9fc
How to Read Extremely Large Text Files in Python | by Gen. Devin DL. | Medium
January 2, 2024 - def chunked_file_reader(fp, block_size): while True: chunk = fp.read(block_size) if not chunk: break yield chunk def read_from_file_v2(filename, block_size=1024*8): with open(filename, 'r') as fp: for chunk in chunked_file_reader(fp, block_size): # processing the content chunk from the file ... AI practitioner & python coder to record what I learned in AI & Data Science & Python project development
🌐
sqlpey
sqlpey.com › python › python-efficiently-read-large-files-in-chunks
Python: Efficiently Read Large Files in Chunks - sqlpey
July 22, 2025 - Python 3.8 introduced the assignment expression (walrus operator :=), enabling a cleaner syntax for reading files in chunks within a while loop.
🌐
pycdlib
clalancette.github.io › pycdlib › example-reading-file-in-chunks.html
Example: Reading a large file in chunks | pycdlib - GitHub Pages
from io import BytesIO import pycdlib iso = pycdlib.PyCdlib() iso.new() foostr = b'foofoo\n' iso.add_fp(BytesIO(foostr), len(foostr), '/FOO.;1') with iso.open_file_from_iso(iso_path='/FOO.;1') as infp: all1 = infp.read() infp.seek(0) first = infp.read(3) second = infp.read() iso.close() Let’s take a closer look at the code. ... As we’ve seen before, import pycdlib. We also import the BytesIO module so we can use a python string as a file-like object.
🌐
DevGenius
blog.devgenius.io › 10-ways-to-work-with-large-files-in-python-effortlessly-handle-gigabytes-of-data-aeef19bc0429
10 Ways to Work with Large Files in Python: Effortlessly Handle Gigabytes of Data! | by Aleksei Aleinikov | Dev Genius
December 1, 2024 - Buffered reading provides a higher level of optimization by processing files in larger internal chunks: with open('large_file.txt', 'rb', buffering=10 * 1024 * 1024) as file: # 10 MB buffer for line in file: process(line) Why use it? Reduces the overhead of frequent disk I/O operations. Memory mapping allows Python to treat a file like a byte array directly in memory.
🌐
Iditect
iditect.com › guide › python › python_howto_read_big_file_in_chunks.html
How to read big file in Python, read big file in chunks, read multiline at each time from a large file, read big file in lazy method
The while loop continues reading chunks until there are no more chunks to read. Both of these methods allow you to read big files in Python without loading the entire file into memory. Choose the approach that best fits your specific use case and file structure. ... Description: Reading large text ...
🌐
DevGlan
devglan.com › python › file-handling-in-python
File Handling in Python
March 20, 2019 - To read a large file in chunk, we can use read() function with while loop to read some chunk data from a text file at a time. with open('test.txt', mode='rt') as f: text = f.read(100) # Reads the first 100 character and moves pointer to 101th ...
Author: Ashish Kumar
Published: 2019
Pages: 674
🌐
GitHub
github.com › danjan1234 › Read-and-process-large-data-sets-in-chunks
GitHub - danjan1234/Read-and-process-large-data-sets-in-chunks: A simple task manager that reads a large data file in chunks and distributes to multiple cores for simultaneous processing
This compact Python module creates a simple task manager for reading and processing large data sets in chunks. The following scenarios are supported: Single file broken into chunks of fixed or variable sizes (chunk size controlled by specific columns)
Starred by 3 users
Forked by 2 users
Languages: Python 100.0% | Python 100.0%
🌐
Stack Overflow
stackoverflow.com › questions › 68422259 › how-to-read-a-large-file-in-chunks
python - How to read a large file in chunks? - Stack Overflow
wordlist_handler = open('passwords.txt') def read_wordlist(wordlist_handler): return wordlist_handler.read(1024) def read_wordlist2(file='passwords.txt'): with open(file , 'r') as f: return list(f) my goal is to iterate over the file similar to the first method
🌐
Oxford Protein Informatics Group
blopig.com › blog › 2016 › 08 › processing-large-files-using-python
Processing large files using python | Oxford Protein Informatics Group
If we process multiple lines of the file at a time as a chunk, we can reduce these operations. The biggest technicality when doing this is noting that when you jump to a location in a file, you are likely not located at the start of a line. For a simple file, as in this example, this just means you need to call readline, which reads to next newline character.
🌐
Zerve
zerve.ai › data-science-problems › pandas › read-large-csv-files-in-chunks
How to Read Large CSV Files in Chunks - Python Pandas
The rule: choose a chunksize that fits in memory (start with 100k rows, adjust based on your column count and dtypes), then process each chunk in a loop. import pandas as pd # ✅ Read in chunks and process iteratively chunks = pd.read_csv(...
🌐
Netalith
netalith.com › blogs › python-tutorial › how-to-read-large-text-files-in-python
Read Large File in Python: Memory-Efficient Methods | Netalith
February 28, 2026 - (python read file without readlines) ... python, iterate over file object line by line). Choose an appropriate chunk_size when streaming binary or files with huge single lines (python read file in chunks 1024 bytes or 4096 bytes ...