Using gzip.GzipFile:
import gzip
with gzip.open('input.gz','rt') as f:
for line in f:
print('got line', line)
Note: gzip.open(filename, mode) is an alias for gzip.GzipFile(filename, mode).
I prefer the former, as it looks similar to with open(...) as f: used for opening uncompressed files.
Using gzip.GzipFile:
import gzip
with gzip.open('input.gz','rt') as f:
for line in f:
print('got line', line)
Note: gzip.open(filename, mode) is an alias for gzip.GzipFile(filename, mode).
I prefer the former, as it looks similar to with open(...) as f: used for opening uncompressed files.
You could use the standard gzip module in python. Just use:
gzip.open('myfile.gz')
to open the file as any other file and read its lines.
More information here: Python gzip module
You can use zcat to stream the uncompressed contents into grep or whatever filter you want, without incurring space overhead. E.g.
zcat bigfile.gz | grep PATTERN_I_NEED > much_smaller_sample
Also, if it's just grep you're streaming to, you can use zgrep e.g.
zgrep PATTERN_I_NEED bigfile.gz > much_smaller_sample
but zgrep doesn't support 100% of the features of grep on some systems.
Decompression takes place in chunks, and you don't need to hold all of the decompressed data in memory to get to a specific line.
You can combine the gzip module with the csv module and process the file row by row:
import gzip
import csv
with gzip.open('googlebooks-eng-all-3gram-20120701-th.gz', 'rb') as fobj:
reader = csv.reader(fobj, delimiter='\t')
for row in reader:
print row
Now you can scan for the rows you want; as long as you don't try to store all rows in a list object but instead process them individually, you won't be using much memory at all.
Quick demo:
>>> import gzip
>>> import csv
>>> fobj = gzip.open('/tmp/googlebooks-eng-all-3gram-20120701-th.gz', 'rb')
>>> reader = csv.reader(fobj, delimiter='\t')
>>> print next(reader)
["T'Hooft , _NOUN_", '1937', '1', '1']
I used the next() function here to get just one row at a time from the reader but the principles are the same as using the reader in a loop.
The above uses very little memory; no more than a few kilobytes in file buffers and the current chunk to decompress, plus the Python strings in the row list.
You can't, because Gzip is not an archive format.
That's a bit of a crap explanation on its own, so let me break this down a bit more than I did in the comment...
Its just compression
Being "just a compression system" means that Gzip operates on input bytes (usually from a file) and outputs compressed bytes. You cannot know whether or not the bytes inside represent multiple files or just a single file -- it is just a stream of bytes that has been compressed. That is why you can accept gzipped data over a network, for example. Its bytes_in -> bytes_out.
What's a manifest?
A manifest is a header within an archive that acts as a table of contents for the archive. Note that now I am using the term "archive" and not "compressed stream of bytes". An archive implies that it is a collection of files or segments that are referred to by a manifest -- a compressed stream of bytes is just a stream of bytes.
What's inside a Gzip, anyway?
A somewhat simplified description of a .gz file's contents is:
- A header with a special number to indicate its a gzip, a version and a timestamp (10 bytes)
- Optional headers; usually including the original filename (if the compression target was a file)
- The body -- some compressed payload
- A CRC-32 checksum at the end (8 bytes)
That's it. No manifest.
Archive formats, on the other hand, will have a manifest inside. That's where the tar library would come in. Tar is just a way to shove a bunch of bits together into a single file, and places a manifest at the front that lets you know the names of the original files and what sizes they were before being concatenated into the archive. Hence, .tar.gz being so common.
There are utilities that allow you to decompress parts of a gzipped file at a time, or decompress it only in memory to then let you examine a manifest or whatever that may be inside. But the details of any manifest are specific to the archive format contained inside.
Note that this is different from a zip archive. Zip is an archive format, and as such contains a manifest. Gzip is a compression library, like bzip2 and friends.
As noted in the other answer, your question can only make sense if I take out the plural: "I have a .gz file and I need to get the name of file inside it using python."
A gzip header may or may not have a file name in it. The gzip utility will normally ignore the name in the header, and decompress to a file with the same name as the .gz file, but with the .gz stripped. E.g. your 1.gz would decompress to a file named 1, even if the header has the file name my_latest_data.json in it. The -N option of gzip will use the file name in the header (as well as the time stamp in the header), if there is one. So gzip -dN 1.gz would create the file my_latest_data.json, instead of 1.
You can find the file name in the header in Python by processing the header manually. You can find the details in the gzip specification.
- Verify that the first three bytes are
1f 8b 08. - Save the fourth byte. Call it
flags. Ifflags & 8is zero, then give up -- there is no file name in the header. - Skip the next six bytes.
- If
flags & 2is not zero, skip two bytes. - If
flags & 4is not zero, then read the next two bytes. Considering them to be in little endian order, make an integer out of those two bytes, calling itxlen. Then skipxlenbytes. - We already know that
flags & 8is not zero, so you are now at the file name. Read bytes until you get to zero byte. Those bytes up to, but not including the zero byte are the file name.
Piping zcat’s output to head -n 1 will decompress a small amount of data, guaranteed to be enough to show the first line, but typically no more than a few buffer-fulls (96 KiB in my experiments):
zcat logfile.gz | head -n 1
Once head has finished reading one line, it closes its input, which closes the pipe, and zcat stops after receiving a SIGPIPE (which happens when it next tries to write into the closed pipe). You can see this by running
(zcat logfile.gz; echo $? >&2) | head -n 1
This will show that zcat exits with code 141, which indicates it stopped because of a SIGPIPE (13 + 128).
You can add more post-processing, e.g. with AWK, to only extract the date:
zcat logfile.gz | awk '{ print $1; exit }'
(On macOS you might need to use gzcat rather than zcat to handle gzipped files.)
You could limit the amount of data you feed to zcat (or gzip -dc), then ask for the first line:
head -c 1000 logfile.gz | zcat 2>/dev/null | head -1 | read logdate otherstuff
Adjust the 1000 if that doesn't capture enough data to get the entire first line.