In Python 3, TemporaryDirectory from the tempfile module can be used.
From the examples:
import tempfile
with tempfile.TemporaryDirectory() as tmpdirname:
print('created temporary directory', tmpdirname)
# directory and contents have been removed
To manually control when the directory is removed, don't use a context manager, as in the following example:
import tempfile
temp_dir = tempfile.TemporaryDirectory()
print(temp_dir.name)
# use temp_dir, and when done:
temp_dir.cleanup()
The documentation also says:
On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem.
At the end of the program, for example, Python will clean up the directory if it wasn't removed, e.g. by the context manager or the cleanup() method. Python's unittest may complain of ResourceWarning: Implicitly cleaning up <TemporaryDirectory... if you rely on this, though.
In Python 3, TemporaryDirectory from the tempfile module can be used.
From the examples:
import tempfile
with tempfile.TemporaryDirectory() as tmpdirname:
print('created temporary directory', tmpdirname)
# directory and contents have been removed
To manually control when the directory is removed, don't use a context manager, as in the following example:
import tempfile
temp_dir = tempfile.TemporaryDirectory()
print(temp_dir.name)
# use temp_dir, and when done:
temp_dir.cleanup()
The documentation also says:
On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem.
At the end of the program, for example, Python will clean up the directory if it wasn't removed, e.g. by the context manager or the cleanup() method. Python's unittest may complain of ResourceWarning: Implicitly cleaning up <TemporaryDirectory... if you rely on this, though.
Use the mkdtemp() function from the tempfile module:
import tempfile
import shutil
dirpath = tempfile.mkdtemp()
# ... do stuff with dirpath
shutil.rmtree(dirpath)
Hi, I'm calling some external software from within python, which basically uses a system call such as os.system("external program -input_file x -output_file y"). For this, I'm using a temporarily created directory (tempfile.mkdtemp) and it works perfectly.
However, I'd like to speed things up and write as little as possible to my hard disk. Therefore I'd like to write into memory, however, I want to maintain a directory and file names that I can use with the os.system call. tempfile.SpooledTemporaryFile does not work for this as far as I can see. Has anybody tried something similar or can help?
Thanks!
If your goal is to speed things up, then this should probably be one of the last things you consider. The OS abstracts things, so that most file operations happen in memory anyway -- the physical writes happen in parallel. So speeding up file writes may not have a measurable effect.
That said, if you are on linux, you could mount a directory with tmpfs, which creates an in memory file system.
Okay, couple of things
-
os.systemis okay, but you should be using thesubprocessmodule, it also gives you a lot more control. You can replaceos.systemwithsubprocess.call("<cmd>", shell=True) -
This is outside the purview of Python, but if you want to keep everything in memory, but still keep filesystem objects, mount a
tmpfsand use that path (Linux). If you're on another platform, I'm not sure what the command would be. -
If you want to avoid the filesystem altogether, if your externally called app supports outputting to stdout, then use
subprocess.Popen(). Eg:
pd = subprocess.Popen('<cmd>', stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
(stdout, stderr) = pd.communicate()
return_code = pd.wait()
To manage resources (like files) in Python, best practice is to use the with keyword, which automatically releases the resources (i.e., cleans up, like closing files); this is available from Python 2.5.
From Python 3.2, you can use tempfile.TemporaryDirectory() instead of tempfile.mkdtmp() – this is usable in with and automatically cleans up the directory:
from tempfile import TemporaryDirectory
with TemporaryDirectory() as temp_dir:
# ... do something with temp_dir
# automatically cleaned up when context exited
If you are using an earlier version of Python (at least 2.5, so have with), you can use backports.tempfile; see Nicholas Bishop’s answer to tempfile.TemporaryDirectory context manager in Python 2.7.
It’s easy and instructive to roll your own class, called a
context manager. The return value of the __enter__() method is bound to the target of the as clause, while the __exit__() method is called when the context is exited – even by exception – and performs cleanup.
import shutil
import tempfile
class TemporaryDirectory(object):
"""Context manager for tempfile.mkdtemp() so it's usable with "with" statement."""
def __enter__(self):
self.name = tempfile.mkdtemp()
return self.name
def __exit__(self, exc_type, exc_value, traceback):
shutil.rmtree(self.name)
You can simplify this with the @contextlib.contextmanager decorator, so you don’t need to write a context manager manually. The code prior to the yield is executed when entering the context, the yielded value is bound to the target of the as, and the code after the yield is executed when exiting the context. This is fundamentally a coroutine that encapsulates the resource acquisition and release, with the yield yielding control to the suite (body) of the with clause. Note that here you do need to have a try...finally block, as @contextlib.contextmanager does not catch exceptions in the yield – this just factors the resource management into a coroutine.
from contextlib import contextmanager
import tempfile
import shutil
@contextmanager
def TemporaryDirectory():
name = tempfile.mkdtemp()
try:
yield name
finally:
shutil.rmtree(name)
As simplylizz notes, if you don’t mind the directory already being deleted (which the above code assumes does not happen), you can catch the “No such file or directory” exception as follows:
import errno
# ...
try:
shutil.rmtree(self.name)
except OSError as e:
# Reraise unless ENOENT: No such file or directory
# (ok if directory has already been deleted)
if e.errno != errno.ENOENT:
raise
You can compare with the standard implementation in tempfile.py; even this simple class has had bugs and evolved over the years.
For background on with, see:
- The Python Tutorial: Methods of File Objects
- With Statement Context Managers
- PEP 343 -- The "with" Statement
Read the documentation, it's simple. ;) From the docs: the directory is readable, writable, and searchable only by the creating user ID.
To delete temp directory try something like this:
import errno
import shutil
import tempfile
try:
tmp_dir = tempfile.mkdtemp() # create dir
# ... do something
finally:
try:
shutil.rmtree(tmp_dir) # delete directory
except OSError as exc:
if exc.errno != errno.ENOENT: # ENOENT - no such file or directory
raise # re-raise exception
Also you can try tempdir package or see its sources.