For some context, I have dozens of csv files in a directory that contain information that I need to process. One of the problems with this though, is that the csv files actually contain several different data sets, each with a different number of columns, column names, column data types, etc. As such, my idea was to preprocess each csv to extract just the lines that contain the data that I need, I can do this by just counting how many columns are in each line of the csv.

My idea was to go through each of the csvs that I need to process, extract the relevant lines from the csvs and write them to a Python NamedTemporaryFile from the tempfile module. Then, once all of the files have had the relevant data extracted, I would then read the data from the temp file into a pandas data frame that I could then work with. However, I keep running into a "Permission denied" error that I'm not entirely sure how to get around. Here is the code (with some sensitive information removed) that I'm working with:

import os
import tempfile
import pandas as pd

if __name__ == '__main__':
    # This is the directory that the csvs are stored in
    dir_path = r'\\My\Private\Directory'

    # get all the csv files and their full paths from the directory 
    files = [os.path.join(dir_path,f) for f in os.listdir(dir_path)]

    # A list of column names for the final pandas dataframe
    # this is just an example list, there are actually 46 columns in total
    columns = ['col1', 'col2']

    # open a named temporary file in the same directory the original csvs came from
    # then loop through all the lines in all the csvs and write the lines with the
    # correct number of columns to the temporary file
    with tempfile.NamedTemporaryFile(dir=dir_path, suffix='.csv', mode='w+') as temp_file:
        for file in files:
            with open(file, 'r') as f:
                for line in f.readlines():
                    if line.count(',') == 46:
                        temp_file.write(line)
        # here I try to read the temp file into the pandas dataframe 
        df = pd.read_csv(temp_file.name, names=columns, header=None, dtype=str)
    
    # However, after trying to read the temp file I get the error:
    # PermissionError: [Errno 13] Permission denied:
    # '\\\\My\\Private\\Directory\\tmps3m6jegs.csv'

    print(df)

As mentioned in the comments in the code block above, when I try the above code, everything seems to work fine up until I try to read the temp file with pandas and get the aforementioned "PermissionError".

In the "NamedTemporaryFile" function, I also tried setting the "delete" parameter to False, which means that the resulting temporary file that is created isn't automatically deleted when the "with" statement ends. When I did this, pandas could read the data from the temp file, but like I said, it doesn't delete the temp file afterwards, which kind of defeats the purpose of the temp file in the first place.

If anyone has any ideas as to what I could be doing wrong or potential fixes I would appreciate the help!

import tempfile
import requests
from PIL import Image
import time

url = "http://i.imgur.com/ANOJOwy.png"
r = requests.get(url) 
with tempfile.NamedTemporaryFile(mode="wb",delete=True) as jpg:
    jpg.write(r.content)
    name = str(jpg.name)
    img = Image.open(name)
    img.show()
    time.sleep(60)

The error:

PermissionError: [Errno 13] Permission denied: 'C:\\Users\\...\\AppData\\Local\\Temp\\tmppgywxdsc'

Edit:

os.startfile(name) works, but I have to choose a program to run it with everytime. Why doesn't PIL work?