TL;DR: if you you need to filter out files/dirs by attributes - use Files.find(), if you don't need to filter by file attributes - use Files.walk().
Details
There is a slight difference which is actually explained in the documentation, but in a way that it feels completely wrong. Reading the source code makes it clear:
Files.find:
return StreamSupport.stream(...) .onClose(iterator::close) .filter(entry -> matcher.test(entry.file(), entry.attributes())) .map(entry -> entry.file());Files.walk:
return StreamSupport.stream(...) .onClose(iterator::close) .map(entry -> entry.file());
This means that if, in your eventual filter, you need to get and validate file attributes - chances are that File.find will be faster. That's because with File.walk, your filter callback will need an extra call to e.g. Files.readAttributes(file, BasicFileAttributes.class), while with File.find - the attributes are already retrieved and given to you in the filter callback.
I just tested it with my sample 10K-files-in-many-folders structure on Windows, by searching files only (i.e. excluding folders):
// pre-Java7/8 way via recursive listFiles (8037 files returned): 1521.657 msec.
for (File f : new File(dir).listFiles()) {
if (f.isDirectory()) {
_getFiles(files, path, pattern);
} else {
...
}
}
// Files.walk(8037 files returned): 1575.766823 msec.
try (Stream<Path> stream = Files.walk(path, Integer.MAX_VALUE) {
files = stream.filter(p -> {
if (Files.isDirectory(p)) { return false; } // this extra check makes it much slower than Files.find
...
}).map(p -> p.toString()).collect(Collectors.toList());
}
// Files.find(8037 files returned): 27.606675 msec.
try (Stream<Path> stream = Files.find(path, Integer.MAX_VALUE, (p, a) -> !a.isDirectory())) {
files = stream.filter(p -> { ... }).map(p -> p.toString()).collect(Collectors.toList());
}
// Files.walkFileTree(8037 returned): 27.443974 msec.
Files.walkFileTree(new File(path).toPath(), new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(Path p, BasicFileAttributes attrs) throws IOException {
...
return FileVisitResult.CONTINUE;
}
});
Answer from Borka on Stack Overflowjava - What is the difference between Files.walk.filter and Files.find? - Stack Overflow
Java Files.Walk to filter files only with starting with "O_" - Stack Overflow
python - Filtering os.walk() dirs and files - Stack Overflow
java - How to use Files.walk()... to get a graph of files based on conditions? - Stack Overflow
TL;DR: if you you need to filter out files/dirs by attributes - use Files.find(), if you don't need to filter by file attributes - use Files.walk().
Details
There is a slight difference which is actually explained in the documentation, but in a way that it feels completely wrong. Reading the source code makes it clear:
Files.find:
return StreamSupport.stream(...) .onClose(iterator::close) .filter(entry -> matcher.test(entry.file(), entry.attributes())) .map(entry -> entry.file());Files.walk:
return StreamSupport.stream(...) .onClose(iterator::close) .map(entry -> entry.file());
This means that if, in your eventual filter, you need to get and validate file attributes - chances are that File.find will be faster. That's because with File.walk, your filter callback will need an extra call to e.g. Files.readAttributes(file, BasicFileAttributes.class), while with File.find - the attributes are already retrieved and given to you in the filter callback.
I just tested it with my sample 10K-files-in-many-folders structure on Windows, by searching files only (i.e. excluding folders):
// pre-Java7/8 way via recursive listFiles (8037 files returned): 1521.657 msec.
for (File f : new File(dir).listFiles()) {
if (f.isDirectory()) {
_getFiles(files, path, pattern);
} else {
...
}
}
// Files.walk(8037 files returned): 1575.766823 msec.
try (Stream<Path> stream = Files.walk(path, Integer.MAX_VALUE) {
files = stream.filter(p -> {
if (Files.isDirectory(p)) { return false; } // this extra check makes it much slower than Files.find
...
}).map(p -> p.toString()).collect(Collectors.toList());
}
// Files.find(8037 files returned): 27.606675 msec.
try (Stream<Path> stream = Files.find(path, Integer.MAX_VALUE, (p, a) -> !a.isDirectory())) {
files = stream.filter(p -> { ... }).map(p -> p.toString()).collect(Collectors.toList());
}
// Files.walkFileTree(8037 returned): 27.443974 msec.
Files.walkFileTree(new File(path).toPath(), new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(Path p, BasicFileAttributes attrs) throws IOException {
...
return FileVisitResult.CONTINUE;
}
});
I believe walk() would be advantageous if you would need to apply some intermediary operation on the directory listing before applying a filter or parallelize the stream.
Really old question, but I still find it when I search for some directories reading ways.
Anyway, assylias commented the first post to add :
filter(p -> p.getFileName().startsWith("O_"))
Which is not working because there is missing the .toString() function. With this you can solve the problem by a more efficient way like it :
Files.walk(Paths.get(SOURCEDIR))
.filter(p -> p.getFileName().toString().startsWith("O_"))
.forEach(System.out::println);
Which one uses the Java 8 Syntax and provides you a standalone line which does the work.
If I understand your question correctly, this should do the work:
Files.walk(Paths.get(SOURCEDIR))
.filter(Files::isRegularFile)
.forEach(filePath -> {
String name = filePath.getFilename().toString();
if (name.startWith("_O")) {
System.out.println(filePath.getFileName());
}
});
This solution uses fnmatch.translate to convert glob patterns to regular expressions (it assumes the includes only is used for files):
import fnmatch
import os
import os.path
import re
includes = ['*.doc', '*.odt'] # for files only
excludes = ['/home/paulo-freitas/Documents'] # for dirs and files
# transform glob patterns to regular expressions
includes = r'|'.join([fnmatch.translate(x) for x in includes])
excludes = r'|'.join([fnmatch.translate(x) for x in excludes]) or r'$.'
for root, dirs, files in os.walk('/home/paulo-freitas'):
# exclude dirs
dirs[:] = [os.path.join(root, d) for d in dirs]
dirs[:] = [d for d in dirs if not re.match(excludes, d)]
# exclude/include files
files = [os.path.join(root, f) for f in files]
files = [f for f in files if not re.match(excludes, f)]
files = [f for f in files if re.match(includes, f)]
for fname in files:
print fname
From docs.python.org:
os.walk(top[, topdown=True[, onerror=None[, followlinks=False]]])
When topdown is True, the caller can modify the dirnames list in-place … this can be used to prune the search …
for root, dirs, files in os.walk('/home/paulo-freitas', topdown=True):
# excludes can be done with fnmatch.filter and complementary set,
# but it's more annoying to read.
dirs[:] = [d for d in dirs if d not in excludes]
for pat in includes:
for f in fnmatch.filter(files, pat):
print os.path.join(root, f)
I should point out that the above code assumes excludes is a pattern, not a full path. You would need to adjust the list comprehension to filter if os.path.join(root, d) not in excludes to match the OP case.
Files.walk(Paths.get("/path/to/stuff/"))
.filter(p -> p.toString().endsWith(".ext"))
.map(p -> p.getParent().getParent())
.distinct()
.forEach(System.out::println);
This filters all files that have the extension and gets the parent path of their directory. distinct ensures that every path is used only once.
You are invoking the method printIfArtifactVersionDirectory for all visited directories. I did a little change to make it obvious:
static void printIfArtifactVersionDirectory(Path path) {
System.out.println("--- " + path);
...
}
With that additional output you will get:
--- C:\Projects\stuff
--- C:\Projects\stuff\org
--- C:\Projects\stuff\org\foo
--- C:\Projects\stuff\org\foo\bar
--- C:\Projects\stuff\org\foo\bar\1.2.3
C:\Projects\stuff\org\foo\bar
--- C:\Projects\stuff\org\foo\bar\1.2.4
C:\Projects\stuff\org\foo\bar
--- C:\Projects\stuff\org\foo\bar\blah
--- C:\Projects\stuff\org\foo\bar\blah\2.1
C:\Projects\stuff\org\foo\bar\blah
--- C:\Projects\stuff\org\foo\bar\blah\2.2
C:\Projects\stuff\org\foo\bar\blah
So you get the output as often as you have artifact version directories. If you want to remember that you already did the output for one directory, you must make store this information somewhere. One quick implementation could be:
static class Foo {
private static final Set<Path> visited = new HashSet<>();
static void printIfArtifactVersionDirectory(Path path) {
...
Path parent = path.getParent();
if (!filePaths.isEmpty() && !visited.contains(parent)) {
visited.add(parent);
System.out.println(parent);
}
}
}
With this you get the expected output:
C:\Projects\stuff\org\foo\bar
C:\Projects\stuff\org\foo\bar\blah
A better solution would be to use the set for storing the visited parents and only print them after visiting them all:
static class PathStore {
private final Set<Path> store = new HashSet<>();
void visit(Path path) {
File f = path.toAbsolutePath().toFile();
List<String> filePaths = Arrays.asList(f.list(new MyExtFilenameFilter()));
if (!filePaths.isEmpty()) {
store.add(path.getParent());
}
}
void print() {
store.forEach(System.out::println);
}
}
Usage:
PathStore pathStore = new PathStore();
Files.walk(Paths.get("/path/to/stuff/"))
.filter(Files::isDirectory)
.forEach(pathStore::visit);
pathStore.print();
I am writing a simple program that walks the file tree to generate various statistics about the files.
For example:
try (Stream<Path> walk = Files.walk(PATH)) {
// Find all directories
List<Path> dirs = walk.filter(Files::isDirectory).collect(Collectors.toList());
// Find all files
List<Path> files = walk.filter(Files::isRegularFile).collect(Collectors.toList());
// Find zip archive files
List<Path> zips = walk.filter(
p -> p.getFileName().toString().toLowerCase().endsWith(".zip"))
.collect(Collectors.toList());
// Find files bigger than 1 Mb
List<Path> filesBiggerThan1Mb = walk.filter(p -> {
try {
return Files.size(p) > 1048576;
} catch (IOException e) {
e.printStackTrace();
return false;
}
}).collect(Collectors.toList());
// Get total size of all files
long totalSize = walk.filter(Files::isRegularFile).mapToLong(p -> {
try {
return Files.size(p);
} catch (IOException e) {
e.printStackTrace();
return 0;
}
}).sum();
}Currently it walks the file tree multiple times by reusing the walk object. Although it seems like either the JRE or os does some caching in memory, and subsequent Files walks are much faster, I am wondering how I can write it in a different way to only need to invoke Files walk only once and do everything in 1 sweep.
This works. It works best to map from Path to File and then see if the list contains the file name.
List<String> fileList = List.of("file1.txt", "file3.dat");
try (Stream<Path> paths = Files.walk(Paths.get(fileSourcePath))) {
paths
.filter(Files::isReadable)
.map(Path::toFile)
.filter(file -> fileList.contains(file.getName()))
.forEach(file->System.out.println(file));
}catch (IOException ex){
ex.printStackTrace();
}
Something like:
file -> fileNameList.contains(file.getName())
?
If you need to sanitize filenames, then, you'd just tack that on. Something like file.getName().toLowerCase(). Or, if there's more you need to do, then maybe define how you want to sanitize/standardize the filenames for comparison, and then call that, like: fileNameList.contains(sanitize(file.getName()))
(caveat: haven't run this)