In Python 3.5 and newer use the new recursive **/ functionality:
configfiles = glob.glob('C:/Users/sam/Desktop/file1/**/*.txt', recursive=True)
When recursive is set, ** followed by a path separator matches 0 or more subdirectories.
In earlier Python versions, glob.glob() cannot list files in subdirectories recursively.
In that case I'd use os.walk() combined with fnmatch.filter() instead:
import os
import fnmatch
path = 'C:/Users/sam/Desktop/file1'
configfiles = [os.path.join(dirpath, f)
for dirpath, dirnames, files in os.walk(path)
for f in fnmatch.filter(files, '*.txt')]
This'll walk your directories recursively and return all absolute pathnames to matching .txt files. In this specific case the fnmatch.filter() may be overkill, you could also use a .endswith() test:
import os
path = 'C:/Users/sam/Desktop/file1'
configfiles = [os.path.join(dirpath, f)
for dirpath, dirnames, files in os.walk(path)
for f in files if f.endswith('.txt')]
Answer from Martijn Pieters on Stack OverflowIn Python 3.5 and newer use the new recursive **/ functionality:
configfiles = glob.glob('C:/Users/sam/Desktop/file1/**/*.txt', recursive=True)
When recursive is set, ** followed by a path separator matches 0 or more subdirectories.
In earlier Python versions, glob.glob() cannot list files in subdirectories recursively.
In that case I'd use os.walk() combined with fnmatch.filter() instead:
import os
import fnmatch
path = 'C:/Users/sam/Desktop/file1'
configfiles = [os.path.join(dirpath, f)
for dirpath, dirnames, files in os.walk(path)
for f in fnmatch.filter(files, '*.txt')]
This'll walk your directories recursively and return all absolute pathnames to matching .txt files. In this specific case the fnmatch.filter() may be overkill, you could also use a .endswith() test:
import os
path = 'C:/Users/sam/Desktop/file1'
configfiles = [os.path.join(dirpath, f)
for dirpath, dirnames, files in os.walk(path)
for f in files if f.endswith('.txt')]
There's a lot of confusion on this topic. Let me see if I can clarify it (Python 3.7):
glob.glob('*.txt') :matches all files ending in '.txt' in current directoryglob.glob('*/*.txt') :same as 1glob.glob('**/*.txt') :matches all files ending in '.txt' in the immediate subdirectories only, but not in the current directoryglob.glob('*.txt',recursive=True) :same as 1glob.glob('*/*.txt',recursive=True) :same as 3glob.glob('**/*.txt',recursive=True):matches all files ending in '.txt' in the current directory and in all subdirectories
So it's best to always specify recursive=True.
What Python version introduced recursive searching with glob?
How does the glob module work for searching files?
Is pathlib preferable over glob?
Do you mean immediate subdirectories, or every directory right down the tree?
Either way, you could use os.walk to do this:
os.walk(directory)
will yield a tuple for each subdirectory. Ths first entry in the 3-tuple is a directory name, so
[x[0] for x in os.walk(directory)]
should give you all of the subdirectories, recursively.
Note that the second entry in the tuple is the list of child directories of the entry in the first position, so you could use this instead, but it's not likely to save you much.
However, you could use it just to give you the immediate child directories:
next(os.walk('.'))[1]
Or see the other solutions already posted, using os.listdir and os.path.isdir, including those at "How to get all of the immediate subdirectories in Python".
Much nicer than the above, because you don't need several os.path.join() and you will get the full path directly (if you wish), you can do this in Python 3.5 and above.
subfolders = [ f.path for f in os.scandir(folder) if f.is_dir() ]
This will give the complete path to the subdirectory.
If you only want the name of the subdirectory use f.name instead of f.path
https://docs.python.org/3/library/os.html#os.scandir
Slightly OT: In case you need all subfolder recursively and/or all files recursively, have a look at this function, that is faster than os.walk & glob and will return a list of all subfolders as well as all files inside those (sub-)subfolders: https://stackoverflow.com/a/59803793/2441026
In case you want only all subfolders recursively:
def fast_scandir(dirname):
subfolders= [f.path for f in os.scandir(dirname) if f.is_dir()]
for dirname in list(subfolders):
subfolders.extend(fast_scandir(dirname))
return subfolders
Returns a list of all subfolders with their full paths. This again is faster than os.walk and a lot faster than glob.
An analysis of all functions
tl;dr:
- If you want to get all immediate subdirectories for a folder use os.scandir.
- If you want to get all subdirectories, even nested ones, use os.walk or - slightly faster - the fast_scandir function above.
- Never use os.walk for only top-level subdirectories, as it can be hundreds(!) of times slower than os.scandir.
- If you run the code below, make sure to run it once so that your OS will have accessed the folder, discard the results and run the test, otherwise results will be screwed.
- You might want to mix up the function calls, but I tested it, and it did not really matter.
- All examples will give the full path to the folder. The pathlib example as a (Windows)Path object.
- The first element of
os.walkwill be the base folder. So you will not get only subdirectories. You can usefu.pop(0)to remove it. - None of the results will use natural sorting. This means results will be sorted like this: 1, 10, 2. To get natural sorting (1, 2, 10), please have a look at https://stackoverflow.com/a/48030307/2441026
Results:
os.scandir took 1 ms. Found dirs: 439
os.walk took 463 ms. Found dirs: 441 -> it found the nested one + base folder.
glob.glob took 20 ms. Found dirs: 439
pathlib.iterdir took 18 ms. Found dirs: 439
os.listdir took 18 ms. Found dirs: 439
Tested with W7x64, Python 3.8.1.
# -*- coding: utf-8 -*-
# Python 3
import time
import os
from glob import glob
from pathlib import Path
directory = r"<insert_folder>"
RUNS = 1
def run_os_walk():
a = time.time_ns()
for i in range(RUNS):
fu = [x[0] for x in os.walk(directory)]
print(f"os.walk\t\t\ttook {(time.time_ns() - a) / 1000 / 1000 / RUNS:.0f} ms. Found dirs: {len(fu)}")
def run_glob():
a = time.time_ns()
for i in range(RUNS):
fu = glob(directory + "/*/")
print(f"glob.glob\t\ttook {(time.time_ns() - a) / 1000 / 1000 / RUNS:.0f} ms. Found dirs: {len(fu)}")
def run_pathlib_iterdir():
a = time.time_ns()
for i in range(RUNS):
dirname = Path(directory)
fu = [f for f in dirname.iterdir() if f.is_dir()]
print(f"pathlib.iterdir\ttook {(time.time_ns() - a) / 1000 / 1000 / RUNS:.0f} ms. Found dirs: {len(fu)}")
def run_os_listdir():
a = time.time_ns()
for i in range(RUNS):
dirname = Path(directory)
fu = [os.path.join(directory, o) for o in os.listdir(directory) if os.path.isdir(os.path.join(directory, o))]
print(f"os.listdir\t\ttook {(time.time_ns() - a) / 1000 / 1000 / RUNS:.0f} ms. Found dirs: {len(fu)}")
def run_os_scandir():
a = time.time_ns()
for i in range(RUNS):
fu = [f.path for f in os.scandir(directory) if f.is_dir()]
print(f"os.scandir\t\ttook {(time.time_ns() - a) / 1000 / 1000 / RUNS:.0f} ms.\tFound dirs: {len(fu)}")
if __name__ == '__main__':
run_os_scandir()
run_os_walk()
run_glob()
run_pathlib_iterdir()
run_os_listdir()
Use glob and filter out all the dirs:
logs = [log for log in glob('/var/opt/server/log/current/*') if not os.path.isdir(log)]
Use the recursive argument to control how deep glob goes down. If you set recursive=False it will only match files in the current directory
Looks like this:
logs = glob('/var/opt/server/log/current/*', recursive=False)
There are a couple of ways:
pathlib.Path().rglob()
Use pathlib.Path().rglob() from the pathlib module, which was introduced in Python 3.5.
from pathlib import Path
for path in Path('src').rglob('*.c'):
print(path.name)
glob.glob()
If you don't want to use pathlib, use glob.glob():
from glob import glob
for filename in glob('src/**/*.c', recursive=True):
print(filename)
For cases where matching files beginning with a dot (.); like files in the current directory or hidden files on Unix based system, use the os.walk() solution below.
os.walk()
For older Python versions, use os.walk() to recursively walk a directory and fnmatch.filter() to match against a simple expression:
import fnmatch
import os
matches = []
for root, dirnames, filenames in os.walk('src'):
for filename in fnmatch.filter(filenames, '*.c'):
matches.append(os.path.join(root, filename))
This version should also be faster depending on how many files you have, as the pathlib module has a bit of overhead over os.walk().
For python >= 3.5 you can use **, recursive=True, i.e.:
import glob
for f in glob.glob('/path/**/*.c', recursive=True):
print(f)
If recursive is
True(defaultFalse), the pattern**will match any files and zero or moredirectoriesandsubdirectories. If the pattern is followed by anos.sep, only directories andsubdirectoriesmatch.
Python 3 Demo