You can do this very simply using pandas.
import pandas as pd
# get only the columns you want from the csv file
df = pd.read_csv(target_path + target_file, usecols=['Column Name1', 'Column Name2'])
result = df.to_dict(orient='records')
Sources:
- pandas.read_csv
- pandas.DataFrame.to_dict
python csv to dictionary using csv or pandas module - Stack Overflow
python - csv.DictReader / csv.DictWriter vs Panda library data Frame - - Stack Overflow
python - Load csv file as a dataframe - Stack Overflow
When using reader or DictReader from the csv module, why do you need to access the return value while the file is still open?
You can do this very simply using pandas.
import pandas as pd
# get only the columns you want from the csv file
df = pd.read_csv(target_path + target_file, usecols=['Column Name1', 'Column Name2'])
result = df.to_dict(orient='records')
Sources:
- pandas.read_csv
- pandas.DataFrame.to_dict
You can use the to_dict method to get a list of dicts:
import pandas as pd
df = pd.read_csv(target_path+target_file, names=Fieldnames)
records = df.to_dict(orient='records')
for row in records:
print row
to_dict documentation:
In [67]: df.to_dict?
Signature: df.to_dict(orient='dict')
Docstring:
Convert DataFrame to dictionary.
Parameters
----------
orient : str {'dict', 'list', 'series', 'split', 'records', 'index'}
Determines the type of the values of the dictionary.
- dict (default) : dict like {column -> {index -> value}}
- list : dict like {column -> [values]}
- series : dict like {column -> Series(values)}
- split : dict like
{index -> [index], columns -> [columns], data -> [values]}
- records : list like
[{column -> value}, ... , {column -> value}]
- index : dict like {index -> {column -> value}}
.. versionadded:: 0.17.0
Abbreviations are allowed. `s` indicates `series` and `sp`
indicates `split`.
Returns
-------
result : dict like {column -> {index -> value}}
File: /usr/local/lib/python2.7/dist-packages/pandas/core/frame.py
Type: instancemethod
unicodecsv library can also be used to read .csv files.
import unicodecsv
import pandas as pd
def read_csv(data):
""" Returns a list of dicts from .csv file passed in as data"""
with open(data, "rb") as f:
reader = list(unicodecsv.DictReader(f))
return reader
file = read_csv('filename.csv') # call the function and pass in your 'filename'
pd.DataFrame(file) # call pd.DataFrame() function with file as an argument
# to convert it to DataFrame object
import pandas as pd
data = pd.read_csv('file_name')
Ok, this will make more sense. Why does this work:
import csv
with open('test.csv') as csv_file:
csv_reader = csv.DictReader(csv_file)
for row in csv_reader:
print(row)But this gives an error that the file is closed:
import csv
with open('test.csv') as csv_file:
csv_reader = csv.DictReader(csv_file)
for row in csv_reader:
print(row)
Traceback (most recent call last): File "c:\Users\John\Documents\Python\test_project\test2.py", line 5, in <module> for row in csv_reader: File "C:\Users\John\AppData\Local\Programs\Python\Python311\Lib\csv.py", line 110, in next self.fieldnames File "C:\Users\John\AppData\Local\Programs\Python\Python311\Lib\csv.py", line 97, in fieldnames self._fieldnames = next(self.reader) ^ ValueError: I/O operation on closed file.
Does DictReader (and also reader) not just run once and return a value? Why is csv_file being accessed after the line it's used in?
Thanks!
csv.reader
Given your desired output, you do not need to use dict or, consequently, csv.DictReader. Instead, just use csv.reader, which returns an iterator. Then use next and list to extract headers and data respectively:
from io import StringIO
import csv
x = StringIO("""play,weather,temperature
yes,sunny,77
no,rainy,60
yes,windy,70""")
# replace x with open('file.csv', 'r')
with x as fin:
reader = csv.reader(fin)
headers = next(reader) # get headers from first row
data = list(reader) # exhaust iterator from second row onwards
The result is a list of headers, and a list of lists for data:
print(headers)
['play', 'weather', 'temperature']
print(data)
[['yes', 'sunny', '77'],
['no', 'rainy', '60'],
['yes', 'windy', '70']]
pandas
If you are willing to use a 3rd party library, Pandas may be a better option as it handles type conversion and indexing more conveniently:
import pandas as pd
df = pd.read_csv('file.csv')
The result is a pd.DataFrame object:
print(df)
play weather temperature
0 yes sunny 77
1 no rainy 60
2 yes windy 70
print(type(df))
<class 'pandas.core.frame.DataFrame'>
DictReader returns a file-like object. It still reads the data from the csv file in one row at a time, but the returned rows are ordered dictionaries instead of lists.
If your files is:
play,weather,temperature
yes,sunny,77
no,rainny,60
yes,windy,70
Then you can use DictReader in the following way:
with open('path/to/file.csv') as fp:
header = fp.readline().strip().split(',')
dreader = DictReader(fp, header)
data = list(dreader)
In this case, data will be a list of OrderedDict objects with the mapping from the headers to each item in the row.
data
#returns:
[OrderedDict([('play', 'yes'), ('weather', 'sunny'), ('temperature', '77')]),
OrderedDict([('play', 'no'), ('weather', 'rainny'), ('temperature', '60')]),
OrderedDict([('play', 'yes'), ('weather', 'windy'), ('temperature', '70')])]