use itertools.chain:

import itertools, gzip

files = ['path2zipfile1', 'path2zipfile2']
it = (gzip.open(f, 'rt') for f in files)

for line in itertools.chain.from_iterable(it):
    print(line)

another version without itertools:

def gen(files):
    for f in files:
        fo = gzip.open(f, 'rt')
        while True:
            line = fo.readline()
            if not line:
                break
            yield line

files = ['path2zipfile1', 'path2zipfile2']
for line in gen(files):
    print(line)

gzip only supports compressing a single file or stream. In your case, the extracted stream is a tar object, so you'd use Python's tarfile library to manipulate the extracted contents. This library actually knows how to cope with .tar.gz so you don't need to explicitly extract the gzip yourself.

First, get the list of all files.

files = ['/path/to/foo.txt.gz.001', '/path/to/foo.txt.gz.002', '/path/to/foo.txt.gz.003']

Then iterate over each file and append to a result file.

with open('./result.gz', 'ab') as result:  # append in binary mode
    for f in files:
        with open(f, 'rb') as tmpf:        # open in binary mode also
            result.write(tmpf.read())

Then extract is using zipfile lib. You could use tempfile to avoid handle with temporary zip file.

You can do this very easily with multiprocessing Pools:

import gzip
import multiprocessing
import shutil

filenames = [
    'a.gz',
    'b.gz',
    'c.gz',
    ...
]

def uncompress(path):
    with gzip.open(path, 'rb') as src, open(path.rstrip('.gz'), 'wb') as dest:
        shutil.copyfileobj(src, dest)

with multiprocessing.Pool() as pool:
    for _ in pool.imap_unordered(uncompress, filenames, chunksize=1):
        pass

This code will spawn a few processes, and each process will extract one file at a time.

Here I've chosen chunksize=1, to avoid stalling processes if some files are bigger than average.

Suppose you list the files in a directory and compress each one. Here is code that builds off one of the gzip examples

import pathlib
import gzip
import shutil

target_dir = pathlib.Path('/home/joe/')
for child in target_dir.iterdir():
    # Skip directories
    if child.is_dir():
        continue
    # You could also check for just .txt files here if desired
    with child.open('rb') as f_in:
        with gzip.open(str(child.with_suffix(".txt.gz")), 'wb') as f_out:
            shutil.copyfileobj(f_in, f_out)

With a directory structure like

/home/joe/
|-- subdir
|-- a.txt
|-- b.txt
|-- c.txt

This snippet will generate

/home/joe
|-- subdir
|-- a.txt
|-- a.txt.gz
|-- b.txt
|-- b.txt.gz
|-- c.txt
|-- c.txt.gz

If you want to remove the original files you could add child.unlink() after the with context.

To put multiple files in a gzipped file, use tarfile.open with a mode of w:gz. Then you can use the addfile method to put serialized objects into it (using a StringIO as the fileobj).

import numpy
np_array_data = numpy.zeros(100)
list_of_strs = ['abc'] * 100

import io
import pickle

np_array_data = io.BytesIO()
numpy.save(np_array_data, np_array)
np_array_data.seek(0)
str_data = io.BytesIO()
pickle.dump(list_of_strs, str_data)
str_data.seek(0)
with tarfile.open('output.tar.gz', mode='w:gz') as dest_file:
    dest_file.addfile(tarfile.TarInfo('np_data'), np_array_data)
    dest_file.addfile(tarfile.TarInfo('str_data'), str_data)

If you only wanted to put a number of numpy arrays into a compressed file, you could just use numpy.savez_compressed.

Since the split files do not need to be readable text files, I would read & write in chunks of bytes, not in lines. This should be faster than reading and writing line by line. We can go with a chunk size of 4,096 bytes because 4,096 or 8,192 is a typical block size on most file systems.

Although, I should note that when I tested your script on my system with a 9 GB text file which has 81 characters per line, it took about 73 seconds to split it into files of 10,000,000 lines each, or about 772 MB per file. The fact that it takes 12 minutes on your laptop likely confirms @Graipher's suspicion that your disk's write speed is the bottleneck here.

Using a refactored version of your script that reads & writes in byte chunks (provided at the end of this review), when I ran it on the same file with --size 772 (split into files of size 772 MB) it took about 35 seconds. Still a decent improvement, but I'd be curious to know how it runs on your machine.

To implement this, I personally like having a helper method like the following which tells us the chunk sizes to read/write:

from typing import Iterator

def chunk_sizes(total_bytes: int, chunk_size: int) -> Iterator[int]:
    for _ in range(total_bytes // chunk_size):
        yield chunk_size

    if remaining_bytes := total_bytes % chunk_size:
        yield remaining_bytes

For example, list(chunk_sizes(10000, 4096)) which returns [4096, 4096, 1808] would tell us that in order to read 10,000 bytes in 4,096 byte chunks, we should read 4,096 bytes, followed by another 4,096 bytes, followed by 1,808 bytes.

General review

• There's no reason to have run() be a staticmethod, and it's weird how FileSplitter creates an instance of itself inside this method.

  If you change the top-level invocation to FileSplitter().run() then run() could be simplified to

  def run(self):
      self.split()
  

  But if you do that you might as well remove run() and make FileSplitter().split() the top-level call. My recommendation is to move the contents of run() under the __name__ == "__main__" guard and then remove run() entirely as it's no longer needed.

• Always open files with a with block to ensure the file is closed properly.

• I recommend using argparse because you get a lot of stuff for free (input type validations, help/usage message with -h, etc.).

• I would move the command-line argument parsing logic out of FileSplitter, because it should ideally only have one responsibility: splitting files.

• Specify the exact exception you want to handle with a try-except. Handling a bare except is considered a bad practice.

• I wouldn't use sys.exit inside methods you might want to test, because it exits the Python interpreter.

• In Python 3, f-strings are a much more legible way of doing string interpolation.

  Before

  new_file_name = "%s.%s" % (self.file_base_name, str(file_number))
  

  After

  new_file_name = f"{self.file_base_name}.{file_number}"
  

Refactored version

#!/usr/bin/env python3

import argparse
import gzip
import os
import itertools

from typing import BinaryIO, Iterator

CHUNK_SIZE = 4096  # bytes


def megabytes_to_bytes(megabytes: int) -> int:
    return megabytes * 1024 * 1024


def chunk_sizes(total_bytes: int, chunk_size: int) -> Iterator[int]:
    for _ in range(total_bytes // chunk_size):
        yield chunk_size

    if remaining_bytes := total_bytes % chunk_size:
        yield remaining_bytes


class FileSplitter:
    input_file_path: str
    output_directory: str
    output_file_base_name: str
    split_size: int  # size of each partition, in MB

    def __init__(
        self, input_file_path: str, output_directory: str, split_size: int
    ) -> None:
        self.input_file_path = input_file_path
        self.output_directory = output_directory
        name = os.path.basename(input_file_path)
        name_without_file_extension = os.path.splitext(name)[0]
        self.output_file_base_name = name_without_file_extension
        self.split_size = split_size

    def get_new_file(self, file_number: int) -> BinaryIO:
        new_file_name = f"{self.output_file_base_name}.{file_number}"
        new_file_path = os.path.join(self.output_directory, new_file_name)
        return open(new_file_path, "wb")

    def split(self) -> None:
        bytes_per_file = megabytes_to_bytes(self.split_size)
        os.makedirs(self.output_directory, exist_ok=True)

        with gzip.open(self.input_file_path, mode="rb") as in_file:
            for file_number in itertools.count(start=1, step=1):
                with self.get_new_file(file_number) as out_file:
                    for chunk_size in chunk_sizes(bytes_per_file, CHUNK_SIZE):
                        chunk = in_file.read(chunk_size)
                        # chunk == "" means we finished reading the input file
                        if not chunk:
                            return
                        out_file.write(chunk)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Read a gzip-compressed file "
        "and split its uncompressed contents into separate files."
    )
    parser.add_argument(
        "file_name", help="a gzip-compressed file, e.g. foo.gz"
    )
    parser.add_argument(
        "--size",
        type=int,
        default=500,
        help="desired size of each partition, in MB",
    )
    args = parser.parse_args()

    output_directory = os.path.join(os.getcwd(), "pychunks")
    splitter = FileSplitter(args.file_name, output_directory, args.size)
    splitter.split()

Yes. Use z = zlib.decompressobj(31), and then use z to decompress until z.unused_data is not empty, or you have processed all of the input. If you get z.unused_data as not empty, then it contains the start of the next gzip stream. Create a new y = zlib.decompressobj object, and start decompression with the contents of z.unused_data, continuing with more data from the file.

This prints the uncompressed size of each concatenated gzip component:

#!/usr/bin/python
import sys
import zlib
z = zlib.decompressobj(31)
count = 0
while True:
    if z.unused_data == "":
        buf = sys.stdin.read(8192)
        if buf == "":
            break
    else:
        print count
        count = 0
        buf = z.unused_data
        z = zlib.decompressobj(31)
    got = z.decompress(buf)
    count += len(got)
print count

Why over engineer? Simply gunzip -r a directory of zipped files:

gunzip -r ./directory_of_zipped_files/

EDIT: Sorry ... now noted that you said windows. This is a *nix solution.

I believe that's because gzip never operates over directories, it acts as a compression algorithm unlike zip and tar where we could compress directories. python's implementation of gzip is to operate on files. However recursive traversal of a directory tree is easy if we look at the os.walk call.

(I haven't tested this)

def gunzip(file_path,output_path):
    with gzip.open(file_path,"rb") as f_in, open(output_path,"wb") as f_out:
        shutil.copyfileobj(f_in, f_out)

def recurse_and_gunzip(root):
    walker = os.walk(root)
    for root,dirs,files in walker:
        for f in files:
            if fnmatch.fnmatch(f,"*.gz"):
                gunzip(f,f.replace(".gz",""))

Your iteration over the file is outside of any try... except, so an exception raised here will terminate the program. If you have a single try...except around the whole thing, then it should work:

    for file in files:
        try:
            with gzip.open(file,'rb') as fin:
                for line in fin:
                    temp = line.decode().split(",")
                    a,b,c,d = temp[0], int(temp[1]), int(temp[2]), int(temp[3])
        except (OSError, ValueError):
            continue

Note also:

• Only catching the specific exceptions that we would expect to occur with a bad file, not other things that should still terminate the program (e.g. KeyboardInterrupt). A bare except: is usually a bad idea.
• It is better to use a with construct with gzip.open