Using the arcname argument of TarFile.add() method is an alternate and convenient way to match your destination.

Example: you want to archive a dir repo/a.git/ to a tar.gz file, but you rather want the tree root in the archive begins by a.git/ but not repo/a.git/, you can do like followings:

archive = tarfile.open("a.git.tar.gz", "w|gz")
archive.add("repo/a.git", arcname="a.git")
archive.close()
Answer from diabloneo on Stack Overflow
🌐
TutorialsPoint
tutorialspoint.com › How-are-files-added-to-a-tar-file-using-Python
How are files added to a tar file using Python?
August 22, 2023 - import tarfile # Create sample ... content for {filename}") # Add multiple files to tar archive with tarfile.open("multi_files.tar", "w") as tar: for file in sample_files: tar.add(file) print("Multiple files added to ...
🌐
Stack Overflow
stackoverflow.com › questions › 20135019 › add-multiple-files-from-multiple-directories-to-unique-tar-gz-archives-using-py
Add multiple files from multiple directories to unique .tar.gz archives using Python - Stack Overflow
import os, tarfile for root, dirs, _ in os.walk('.'): for d in dirs: with tarfile.open(d + '.tar.gz', 'w:gz') as tar: for _, _, files in os.walk(os.path.join(root, d)): for f in files: tar.add(os.path.join(root, d, f), arcname=f) ... A couple ...
🌐
AskPython
askpython.com › home › the tarfile module – how to work with tar files in python?
The tarfile Module - How to work with tar files in Python? - AskPython
February 16, 2023 - The following code is an implementation for creating a tar file in Python. Here we use open() method for creating tar file and add() method for adding other files to a tar file. #import module import tarfile #declare filename filename= ...
🌐
TutorialsPoint
tutorialspoint.com › article › read-and-write-tar-archive-files-using-python-tarfile
Read and write tar archive files using Python (tarfile)
import tarfile, glob >>> fp=tarfile.open('file.tar','w') >>> for file in glob.glob('*.*'): fp.add(file) >>> fp.close() Creation and extraction of tar files can be achieved through command line interface. For example ‘lines.txt’ file is added in a tar file by following command executed in command window · C:\python36 >python -m tarfile -c line.tar lines.txt
Top answer
1 of 3
2

A python solution to do the job completely. It creates a tar.gz file from the latest files of all sub directories of a given directory.

The script

#!/usr/bin/env python3

import os
import time
import tarfile

files_dir = "/path/to/directory/with/subdirectories"
targeted_file = "/path/to/latest_files.tar.gz"

latest_files = []
# get the latest files of all sub directories
for root, dirs, files in os.walk(files_dir):
    for dr in dirs:
        dr = root+"/"+dr
        filelist = []
        for item in os.listdir(dr):
            file = dr+"/"+item
            if os.path.isfile(file):
                filelist.append((file, os.stat(file).st_mtime))
        filelist.sort(key=lambda x: x[1])
        if len(filelist) != 0:
            latest_files.append(filelist[-1][0])
# write to latest_files.tar.gz
tar = tarfile.open(targeted_file, "w:gz")
for file in latest_files:
    tar.add(file, arcname = file.split("/")[-1])
tar.close()

Copy the script into an empty file, set in the head section of the script the directory, containing the sub directories (files_dir =), and the path to the tar.gz file (targeted_file =), and save it as get_latest.py.

Run it by the command:

python3 /path/to/get_latest.py

What it does

The script first lists all (sub-) directories. Within the sub directories, it lists the files, sorts them by modification date and adds the latest ones to the "master" list, to be included in the compressed file.


To set the number of versions to include

To give the answer a more universal character, below a version of the script in which you can set the numbers of (latest) versions to include in the compressed file. If the number of set versions (versions =) exceeds the actual number of files in a folder, all files are included.

The script

#!/usr/bin/env python3

import os
import time
import tarfile

files_dir = "/path/to/directory/with/subdirectories"
targeted_file = "/path/to/latest_files.tar.gz"
versions = 1

latest_files = []
# get the latest files of subdirectories
for root, dirs, files in os.walk(files_dir):
    for dr in dirs:
        dr = root+"/"+dr
        filelist = []
        for item in os.listdir(dr):
            file = dr+"/"+item
            if os.path.isfile(file):
                filelist.append((file, os.stat(file).st_mtime))
        filelist.sort(key=lambda x: x[1])
        if len(filelist) != 0:
            for item in filelist[-versions:]:
                latest_files.append(item[0])
# write to latest_files.tar.gz
tar = tarfile.open(targeted_file, "w:gz")
for file in latest_files:
    tar.add(file, arcname = file.split("/")[-1])
tar.close()
2 of 3
1

"Latest" is hard to define. I think you can use find in two ways here:

  1. List all files modified later than another file:

    find . -type f -newer b/1
    
  2. List all files modified less than x minutes ago (say 10):

    find . -type f -mmin -10
    

Take your pick.

🌐
Python Engineer
python-engineer.com › posts › tarfile-python
How to work with tarball/tar files in Python - Python Engineer
TAR stands for Tape Archive Files and this format is used to bundle a set of files into a single file, this is specifically helpful when archiving older files or sending a bunch of files over the network. ... The Python programming language has tarfile standard module which can be used to work with tar files with support for gzip, bz2, and lzma compressions.
🌐
Stack Overflow
stackoverflow.com › questions › 78296217 › adding-multiple-files-to-tar-archives-root-directory-without-creating-an-enclos
python - Adding multiple files to TAR archive's root directory without creating an enclosing directory - Stack Overflow
In other words, when I extract the .tgz file, I want to see file1.txt and file2.txt (not the parent directory myfiles) at the root of the archive. When I try to accomplish this using python's built-in tarfile package, I do the following in order: Use tarfile.open("myArchive.tgz", "w:gz") to initialize the archive and create the TarFile object · Use TarFile.add("myfiles/file1.txt", "file1.txt") to add one file to the TarFile object
🌐
DNMTechs
dnmtechs.com › adding-files-to-tarfile-without-directory-hierarchy-in-python-3
Adding Files to Tarfile Without Directory Hierarchy in Python 3 – DNMTechs – Sharing and Storing Technology Knowledge
By specifying the arcname parameter when adding files, you can control the name of the file within the tarfile without including the directory hierarchy. This can be useful when you want to create a flat tarfile or maintain a specific structure within the tarfile.
Find elsewhere
Top answer
1 of 2
14

From tarfile documentation:

Note that 'a:gz' or 'a:bz2' is not possible. If mode is not suitable to open a certain (compressed) file for reading, ReadError is raised. Use mode 'r' to avoid this. If a compression method is not supported, CompressionError is raised.

So I guess you should decompress it using gzip library, add the files using the a: mode in tarfile, and then compress again using gzip.

2 of 2
6

David Dale asks:

Update. From the documentation, it follows that gz files cannot be open in a mode. If so, what is the best way to add or update files in an existing archive?

Short answer:

  1. decompress / unpack archive
  2. replace / add file(s)
  3. repack / compress archive

I tried to do it in memory using gzip's and tarfile's and file/stream interfaces but did not manage to get it running - the tarball has to be rewritten anyway, since replacing a file is apparently not possible. So it's better to just unpack the whole archive.

Wikipedia on tar, gzip.

The script, if run directly, also tries to generates the test images "a.png, b.png, c.png, new.png" (requiring Pillow) and the initial archive "test.tar.gz" if they don't exist. It then decompresses the archive into a temporary directory, overwrites "a.png" with the contents of "new.png", and packs all files, overwriting the original archive. Here are the individual files:


Of course the script's functions can also be run sequentially in interactive mode, in order to have a chance to look at the files. Assuming the script's filename is "t.py":

>>> from t import *
>>> make_images()
>>> make_archive()
>>> replace_file()
Workaround

Here we go (the essential part is in replace_file()):

#!python3
#coding=utf-8
"""
Replace a file in a .tar.gz archive via temporary files
   https://stackoverflow.com/questions/28361665/how-to-append-a-file-to-a-tar-file-use-python-tarfile-module
"""

import sys        #
import pathlib    # https://docs.python.org/3/library/pathlib.html
import tempfile   # https://docs.python.org/3/library/tempfile.html
import tarfile    # https://docs.python.org/3/library/tarfile.html
#import gzip      # https://docs.python.org/3/library/gzip.html

gfn = "test.tar.gz"
iext = ".png"

replace = "a"+iext
replacement = "new"+iext

def make_images():
    """Generate 4 test images with Pillow (PIL fork, http://pillow.readthedocs.io/)"""
    try:
        from PIL import Image, ImageDraw, ImageFont
        font = ImageFont.truetype("arial.ttf", 50)

        for k,v in {"a":"red", "b":"green", "c":"blue", "new":"orange"}.items():
            img = Image.new('RGB', (100, 100), color=v)
            d = ImageDraw.Draw(img)
            d.text((0, 0), k, fill=(0, 0, 0), font=font)
            img.save(k+iext)
    except Exception as e:
        print(e, file=sys.stderr)
        print("Could not create image files", file=sys.stderr)
        print("(pip install pillow)", file=sys.stderr)

def make_archive():
    """Create gzip compressed tar file with the three images"""
    try:
        t = tarfile.open(gfn, 'w:gz')
        for f in 'abc':
            t.add(f+iext)
        t.close()
    except Exception as e:
        print(e, file=sys.stderr)
        print("Could not create archive", file=sys.stderr)

def make_files():
    """Generate sample images and archive"""
    mi = False
    for f in ['a','b','c','new']:
        p = pathlib.Path(f+iext)
        if not p.is_file():
            mi = True
    if mi:
        make_images()
    if not pathlib.Path(gfn).is_file():
        make_archive()

def add_file_not():
    """Might even corrupt the existing file?"""
    print("Not possible: tarfile with \"a:gz\" - failing now:", file=sys.stderr)
    try:
        a = tarfile.open(gfn, 'a:gz')  # not possible!
        a.add(replacement, arcname=replace)
        a.close()
    except Exception as e:
        print(e, file=sys.stderr)

def replace_file():
    """Extract archive to temporary directory, replace file, replace archive """
    print("Workaround", file=sys.stderr)

    # tempdir
    with tempfile.TemporaryDirectory() as td:
        # dirname to Path
        tdp = pathlib.Path(td)

        # extract archive to temporry directory
        with tarfile.open(gfn) as r:
            r.extractall(td)

        # print(list(tdp.iterdir()), file=sys.stderr)

        # replace target in temporary directory
        (tdp/replace).write_bytes( pathlib.Path(replacement).read_bytes() )

        # replace archive, from all files in tempdir
        with tarfile.open(gfn, "w:gz") as w:
            for f in tdp.iterdir():
                w.add(f, arcname=f.name)
    #done

def test():
    """as the name suggests, this just runs some tests ;-)"""
    make_files()
    #add_file_not()
    replace_file()

if __name__ == "__main__":
    test()

If you want to add files instead of replacing them, obviously just omit the line that replaces the temporary file, and copy the additional files into the temp directory. Make sure that pathlib.Path.iterdir then also "sees" the new files to be added to the new archive.


I've put this in a somewhat more useful function:

def targz_add(targz=None, src=None, dst=None, replace=False):
    """Add <src> file(s) to <targz> file, optionally replacing existing file(s).
    Uses temporary directory to modify archive contents.
    TODO: complete error handling...
    """
    import sys, pathlib, tempfile, tarfile

    # ensure targz exists
    tp = pathlib.Path(targz)
    if not tp.is_file():
        sys.stderr.write("Target '{}' does not exist!\n".format(tp) )
        return 1

    # src path(s)
    if not src:
        sys.stderr.write("No files given.\n")
        return 1
    # ensure iterable of string(s)
    if not isinstance(src, (tuple, list, set)):
        src = [src]
    # ensure path(s) exist
    srcp = []
    for s in src:
        sp = pathlib.Path(s)
        if not sp.is_file():
            sys.stderr.write("Source '{}' does not exist.\n".format(sp) )
        else:
            srcp.append(sp)

    if not srcp:
        sys.stderr.write("None of the files exist.\n")
        return 1

    # dst path(s) (filenames in archive)
    dstp = []
    if not dst:
        # default: use filename only
        dstp = [sp.name for sp in srcp]
    else:
        if callable(dst):
            # map dst to each Path, ensure results are Path
            dstp = [pathlib.Path(c) for c in map(dst, srcp)]
        elif not isinstance(dst, (tuple, list, set)):
            # ensure iterable of string(s)
            dstp = [pathlib.Path(dst).name]
        elif isinstance(dst, (tuple, list, set)):
            # convert each string to Path
            dstp = [pathlib.Path(d) for d in dst]
        else:
            # TODO directly support iterable of (src,dst) tuples
            sys.stderr.write("Please fix me, I cannot handle the destination(s) '{}'\n".format(dst) )
            return 1

    if not dstp:
        sys.stderr.write("None of the files exist.\n")
        return 1

    # combine src and dst paths
    sdp = zip(srcp, dstp) # iterator of tuples

    # temporary directory
    with tempfile.TemporaryDirectory() as tempdir:
        tempdirp = pathlib.Path(tempdir)

        # extract original archive to temporry directory
        with tarfile.open(tp) as r:
            r.extractall(tempdirp)

        # copy source(s) to target in temporary directory, optionally replacing it
        for s,d in sdp:
            dp = tempdirp/d

            # TODO extend to allow flag individually
            if not dp.is_file or replace:
                sys.stderr.write("Writing '{1}' (from '{0}')\n".format(s,d) )
                dp.write_bytes( s.read_bytes() )
            else:
                sys.stderr.write("Skipping '{1}' (from '{0}')\n".format(s,d) )

        # replace original archive with new archive from all files in tempdir
        with tarfile.open(tp, "w:gz") as w:
            for f in tempdirp.iterdir():
                w.add(f, arcname=f.name)

    return None

And a few "tests" as example:

# targz_add("test.tar.gz", "new.png", "a.png")
# targz_add("test.tar.gz", "new.png", "a.png", replace=True)
# targz_add("test.tar.gz", ["new.png"], "a.png")
# targz_add("test.tar.gz", "new.png", ["a.png"], replace=True)
targz_add("test.tar.gz", "new.png", lambda x:str(x).replace("new","a"), replace=True)

shutil also supports archives, but not adding files to one:

https://docs.python.org/3/library/shutil.html#archiving-operations

New in version 3.2.
Changed in version 3.5: Added support for the xztar format.
High-level utilities to create and read compressed and archived files are also provided. They rely on the zipfile and tarfile modules.


Here's adding a file by extracting to memory using io.BytesIO, adding, and compressing:

import io
import gzip
import tarfile

gfn = "test.tar.gz"
replace = "a.png"
replacement = "new.png"

print("reading {}".format(gfn))
m = io.BytesIO()
with gzip.open(gfn) as g:
    m.write(g.read())

print("opening tar in memory")
m.seek(0)
with tarfile.open(fileobj=m, mode="a") as t:
    t.list()
    print("adding {} as {}".format(replacement, replace))
    t.add(replacement, arcname=replace)
    t.list()

print("writing {}".format(gfn))
m.seek(0)
with gzip.open(gfn, "wb") as g:
    g.write(m.read())

it prints

reading test.tar.gz
opening tar in memory
?rw-rw-rw- 0/0        877 2018-04-11 07:38:57 a.png 
?rw-rw-rw- 0/0        827 2018-04-11 07:38:57 b.png 
?rw-rw-rw- 0/0        787 2018-04-11 07:38:57 c.png 
adding new.png as a.png
?rw-rw-rw- 0/0        877 2018-04-11 07:38:57 a.png 
?rw-rw-rw- 0/0        827 2018-04-11 07:38:57 b.png 
?rw-rw-rw- 0/0        787 2018-04-11 07:38:57 c.png 
-rw-rw-rw- 0/0       2108 2018-04-11 07:38:57 a.png 
writing test.tar.gz

Optimizations are welcome!

🌐
GitHub
github.com › python › cpython › issues › 116931
tarfile.py: TarFile.addfile not adding all files · Issue #116931 · python/cpython
March 17, 2024 - import tarfile archive = tarfile.open("test1.tar", mode="w", format=tarfile.GNU_FORMAT) archive.addfile(archive.gettarinfo(name="test1/A")) archive.addfile(archive.gettarinfo(name="test1/B")) archive.close() print(tarfile.open("test1.tar", mode="r").getnames()) Expected output: ['test1/A', 'test1/B'] Returned output: ['test1/A'] Reproduced on these Python versions: Python 3.11.6 (main, Nov 28 2023, 09:22:32) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux · 3.11 · macOS · gh-116931: Add fileobj parameter check for Tarfile.addfile #117988 ·
Author   python
🌐
Python
docs.python.org › 3 › library › tarfile.html
tarfile — Read and write tar archive files
Source code: Lib/tarfile.py The tarfile module makes it possible to read and write tar archives, including those using gzip, bz2 and lzma compression. Use the zipfile module to read or write.zip fi...
🌐
Python Module of the Week
pymotw.com › 2 › tarfile
tarfile – Tar archive access - Python Module of the Week
February 22, 2009 - $ python tarfile_extractall_members.py ['README.txt'] To create a new archive, simply open the TarFile with a mode of 'w'. Any existing file is truncated and a new archive is started. To add files, use the add() method.
🌐
Real Python
realpython.com › ref › stdlib › tarfile
tarfile | Python Standard Library – Real Python
Language: Python · >>> with tarfile.open("new_archive.tar", mode="w") as tar_file: ... tar_file.add("file1.txt") ... tar_file.add("file2.txt") ... Archiving and compressing multiple files or directories into a single tar archive · Extracting specific files or all contents from tar archives ·
🌐
Coderz Column
coderzcolumn.com › tutorials › python › tarfile-simple-guide-to-work-with-tape-archives-in-python
tarfile - Simple Guide to Work with Tape Archives in Python by Sunny Solanki
October 4, 2022 - Extracting Files to Folder : zen_of_python_images2 ... total 16 -rw-rw-r-- 1 sunny sunny 6471 Jan 16 09:56 dr_apj_kalam.jpeg -rw-rw-r-- 1 sunny sunny 5015 Mar 12 20:45 intro_ball.gif · Our code for this example is almost the same as our code from previous examples. We loop through each member of the archive and retrieve their contents using extractfile() method. We then write them by creating a file. We are also comparing the contents of the file at the end. import tarfile import os ### Creating archive compressed_file = tarfile.open(name="zen_of_python3.tar", mode="r") folder_name = compressed_file.name[:-4] os.makedirs(folder_name, exist_ok=True) for member_name in compressed_file.getnames(): file_content = compressed_file.extractfile(member_name) with open(os.path.join(folder_name,member_name), "wb") as fp: fp.write(file_content.read()) compressed_file.close()
🌐
TheLinuxCode
thelinuxcode.com › home › an in-depth 2500+ word guide to mastering tarfiles in python
An In-Depth 2500+ Word Guide to Mastering Tarfiles in Python – TheLinuxCode
December 27, 2023 - The tarfile module contains a robust set of features beyond just reading and writing tarballs. Here are some additional capabilities and expert tips for working with tarfiles in Python: To add files to an existing tarball, open it in ‘a‘ append mode:
🌐
LinuxConfig
linuxconfig.org › home › how to create and manipulate tar archives using python
Use Tarfile Module in Python: A Complete Guide
June 3, 2020 - The TarInfo object doesn’t contain the actual file data. Some of the attributes of the TarInfo object are: ... In this tutorial we learned the basic usage of the tarfile Python module, and we saw how we can use it to work with tar archives. We saw the various operating modes, what the TarFile and TarInfo classes represent, and some of the most used methods to list the content of an archive, to add new files or to extract them.
🌐
Tutorialspoint
tutorialspoint.com › python › tarfile_module.htm
The tarfile Module
import tarfile, glob fp=tarfile.open('file.tar','w') for file in glob.glob('*.*'): fp.add(file) fp.close() Creation and extraction of tar files can be achieved through command line interface. For example 'lines.txt' file is added in a tar file by following command executed in command window − · C:\python311 >python -m tarfile -c line.tar lines.txt