I meet this problem as well, and list the complete example based on ekhumoro's answer
import os, tarfile
output_dir = "."
tar = tarfile.open(tar_file)
for member in tar.getmembers():
if member.isreg(): # skip if the TarInfo is not files
member.name = os.path.basename(member.name) # remove the path by reset it
tar.extract(member,output_dir) # extract
I meet this problem as well, and list the complete example based on ekhumoro's answer
import os, tarfile
output_dir = "."
tar = tarfile.open(tar_file)
for member in tar.getmembers():
if member.isreg(): # skip if the TarInfo is not files
member.name = os.path.basename(member.name) # remove the path by reset it
tar.extract(member,output_dir) # extract
The data attributes of a TarInfo object are writable. So just change the name to whatever you want and then extract it:
import sys, os, tarfile
args = sys.argv[1:]
tar = tarfile.open(args[0])
member = tar.getmember(args[1])
member.name = os.path.basename(member.name)
path = args[2] if len(args) > 2 else ''
tar.extract(member, path)
Extract only a single directory from tar (in python) - Stack Overflow
Python: Untar file and change directory to extracted folder
python - Extract all files with directory path in given directory - Stack Overflow
Extract file from directory inside tar file without creating that directory - Unix & Linux Stack Exchange
Building on the second example from the tarfile module documentation, you could extract the contained sub-folder and all of its contents with something like this:
with tarfile.open("sample.tar") as tar:
subdir_and_files = [
tarinfo for tarinfo in tar.getmembers()
if tarinfo.name.startswith("subfolder/")
]
tar.extractall(members=subdir_and_files)
This creates a list of the subfolder and its contents, and then uses the recommended extractall() method to extract just them. Of course, replace "subfolder/" with the actual path (relative to the root of the tar file) of the sub-folder you want to extract.
The other answer will retain the subfolder path, meaning that subfolder/a/b will be extracted to ./subfolder/a/b. To extract a subfolder to the root, so subfolder/a/b would be extracted to ./a/b, you can rewrite the paths with something like this:
def members(tf):
l = len("subfolder/")
for member in tf.getmembers():
if member.path.startswith("subfolder/"):
member.path = member.path[l:]
yield member
with tarfile.open("sample.tar") as tar:
tar.extractall(members=members(tar))
Looks like you may have already found an answer, but here's my version anyway:
import sys, tarfile
def get_members(tar, prefix):
if not prefix.endswith('/'):
prefix += '/'
offset = len(prefix)
for tarinfo in tar.getmembers():
if tarinfo.name.startswith(prefix):
tarinfo.name = tarinfo.name[offset:]
yield tarinfo
args = sys.argv[1:]
if len(args) > 1:
tar = tarfile.open(args[0])
path = args[2] if len(args) > 2 else '.'
tar.extractall(path, get_members(tar, args[1]))
with tarfile.open('sourcefile.tgz', 'r:gz') as _tar:
for member in _tar:
if member.isdir():
continue
fname = member.name.rsplit('/',1)[1]
_tar.makefile(member, 'desination_dir' + '/' + fname)
POSIXly (except for the gunzip part that is not a POSIX command):
gunzip < file.tar.gz | pax -rs'|.*/||' dir1/file1
With bsdtar:
tar -s'|.*/||' -xf file.tar.gz dir1/file1
With GNU tar:
tar --transform='s|.*/||' -xf file.tar.gz dir1/file1
With star:
star -s '|*/||' -x -f file.tar.gz dir1/file1
Note that for some of those, if the file is a a symlink, the substitution will also affect the target of the symlink.
You could extract to stdout (-O) and pipe it to the wanted filename.
Version: Python 2.7
OS: MacOS Mojave
IDE: Pycharm Community 2019.2
I'm having trouble downloading tar.gz files from `pypi.org/project` and unzipping them. The use case for this is that we can't use any actual package management, so we have to manually put packages where we need them in the local folders. My solution to this is to read a requirements file and pull the tar.gz and .zip files for the given versions and then write to local files.
I've got it working for zip files, and it works exactly how I want it to. Zip files are handled by the 'elif' statement, tar files are handled by the 'if' statement. For some reason, for the tar file links I pass in, no directories are created, no files are extracted.
I did a test by putting the path to my local copy of a tar file directly in this line `tarData = tarfile.open(fileobj=zipData, mode='r:*')` instead of `zipData` and it worked, so I think it has something to do with the format the file is being either downloaded in or handled in StringIO, but I'm not getting any exceptions so I have no way to narrow down what the issue could be.
Below is the slice of code I'm using to unzip these files:
import os
import bs4 as bs
import urllib2
import re
from StringIO import StringIO
import zipfile
import tarfile
import requests
def install_packages_locally(compressed_link, directory):
file_name = '_'.join(compressed_link.split('/')[-1].split('-')[:-1])
file_ext = compressed_link.split('-')[-1]
response = requests.get(compressed_link, stream=True)
if file_name in os.listdir(directory):
print('Skipping %s' % file_name)
return True
zipData = StringIO()
if file_ext.endswith('.tar.gz'):
zipData.write(response.raw.read())
print('writing %s to %s' % (file_name, directory))
tarData = tarfile.open(fileobj=zipData, mode='r:*')
tarData.extractall(path='/dir/Packages')
tarData.close()
zipData.close()
return True
elif file_ext.endswith('.zip'):
zipData.write(response.content)
print('writing %s to %s' % (file_name, directory))
unzipData = zipfile.ZipFile(zipData)
unzipData.extractall(path='/dir/Packages/')
unzipData.close()
zipData.close()
return TrueUsing the arcname argument of TarFile.add() method is an alternate and convenient way to match your destination.
Example: you want to archive a dir repo/a.git/ to a tar.gz file, but you rather want the tree root in the archive begins by a.git/ but not repo/a.git/, you can do like followings:
archive = tarfile.open("a.git.tar.gz", "w|gz")
archive.add("repo/a.git", arcname="a.git")
archive.close()
You can use tarfile.addfile(), in the TarInfo object, which is the first parameter, you can specify a name that's different from the file you're adding.
This piece of code should add /path/to/filename to the TAR file but will extract it as myfilename:
tar.addfile(tarfile.TarInfo("myfilename.txt"), open("/path/to/filename.txt"))
Using help from kenny's comment, I did what I wanted to do by parsing the data i got from urlopen, use BytesIO, and use that as the fileobj argument for tarfile.open:
from urllib.request import urlopen
import tarfile
from io import BytesIO
r = urlopen("https://url/file.tar.gz")
t = tarfile.open(name=None, fileobj=BytesIO(r.read()))
t.extractall("/somedirectory/")
t.close()
without writing TAR file to disk, you could use python subprocess module to run shell commands for you:
import subprocess
# some params
shell_cmd = 'curl -L "https://somewebsite.com/file.tar.gz" | tar xzf -'
i_trust_this_string_cmd = True
throw_error_on_fail = True
timeout_after_seconds = 10 # or None
convert_output_from_bytes_to_string = True
#
# run shell as subprocesses to this one and get results
cp = subprocess.run(
[shell_cmd],
shell=i_trust_this_string_cmd,
check=throw_error_on_fail,
timeout=timeout_after_seconds,
text=convert_output_from_bytes_to_string
)
#status_code = cp.returncode
try:
cp.check_returncode() # triggers exceptions if errors occurred
print(cp.stdout) # if you want to see the output (text in this case)
except subprocess.CalledProcessError as cpe:
print(cpe)
except subprocess.TimeoutExpired as te:
print(te)
If you wanted more control, you could provide a PIPE for say STDOUT, STDERR e.g.
with open('/tmp/stdout.txt', 'w+') as stdout:
with open('/tmp/stderr.txt', 'w+') as stderr:
cp = subprocess.run([...], stdout=stdout, stderr=stderr)
...