The pattern rules for glob are not regular expressions. Instead, they follow standard Unix path expansion rules. There are only a few special characters: two different wild-cards, and character ranges are supported [from pymotw: glob – Filename pattern matching].
So you can exclude some files with patterns.
For example to exclude manifests files (files starting with _) with glob, you can use:
files = glob.glob('files_path/[!_]*')
The pattern rules for glob are not regular expressions. Instead, they follow standard Unix path expansion rules. There are only a few special characters: two different wild-cards, and character ranges are supported [from pymotw: glob – Filename pattern matching].
So you can exclude some files with patterns.
For example to exclude manifests files (files starting with _) with glob, you can use:
files = glob.glob('files_path/[!_]*')
You can deduct sets and cast it back as a list:
list(set(glob("*")) - set(glob("eph*")))
python - Excluding files in glob.glob() - Stack Overflow
regex - python glob for "excluding pattern" - Stack Overflow
python - How to exclude some files by name using glob.glob("")? - Stack Overflow
How do I exclude a list of files from os.walk?
What if I have nested directories?
How do I implement advanced filtering conditions?
Use glob and filter out all the dirs:
logs = [log for log in glob('/var/opt/server/log/current/*') if not os.path.isdir(log)]
Use the recursive argument to control how deep glob goes down. If you set recursive=False it will only match files in the current directory
Looks like this:
logs = glob('/var/opt/server/log/current/*', recursive=False)
Ones you have your list of files you can use list comprehension to filter and remove any files that contain 'ER'.
files = [f for f in files if 'ER' not in f]
There are other libraries that can do exclusions. For instance wcmatch (full disclosure, I'm the author of it) allows you to use exclusion patterns (when enabled via a flag). Exclusion patterns are given along with a normal patterns, and it will filter the returned list of files:
from wcmatch import glob
glob.glob(['*.txt', '!*ER*.txt'], flags=glob.N)
Here is a real world example:
from wcmatch import glob
>>> glob.glob(['*.md'], flags=glob.N)
['LICENSE.md', 'README.md']
>>> glob.glob(['*.md', '!*EA*.md'], flags=glob.N)
['LICENSE.md']
you are using the wrong character to say none of this character...
If you're looking to find any file which has log_ at the beginning, and then a load of characters where none of them are _, then you just need to do this:
log_[^_]*
You are close. The pattern you are looking for is log_[^_]*. It says it must have 'log_' followed by zero or more non-underscore characters.
You can make the pattern less exclusive as some comments mention by doing something like glob.glob('*_*_*_*.json'). More details can be found here https://docs.python.org/3.5/library/glob.html#glob.glob.
This isn't ever clean and since glob isn't regular regex it isn't very expressive. Since ordering doesn't seem very important you could do something like
excludedFiles = ['merged_file.json']
includedFiles = glob.glob('*.json')
# other code here
print list(set(includedFiles) - set(excludedFile))
That answers your question however I think a better solution to your problem is separate your raw data and generated files into different directories. I think that's generally a good practice when you're doing adhoc work with data.
If you want to remove only the latest file added, then you can try this code.
import os
import glob
jsonFiles = []
jsonPattern = os.path.join('*.json')
fileList = glob.glob(jsonPattern)
for file in fileList:
jsonFiles.append(file)
print jsonFiles
latestFile = max(jsonFiles, key=os.path.getctime)
print latestFile
jsonFiles.remove(latestFile)
print jsonFiles
Output:
['man_pro_desc_nordstrom.json', 'man_pro_desc_Zalando.json', 'men_pro_desc_zalora.json', 'merged_file.json']
merged_file.json
['man_pro_desc_nordstrom.json', 'man_pro_desc_Zalando.json', 'men_pro_desc_zalora.json']
I've tried searching for this, but I could only find how to exclude dirs from the search, and the only instance of how to exclude a file is with file type. Am I missing something here?
My code
ignore_list = ['Item 1', 'Item 2', ....]
for root, dirs, files in os.walk(directory):
files[:] = [f for f in files if f not in ignore_list]
for name in files:
if any(letter.isupper() for letter in name):
print('file' + name + ' is uppercased')