Pandas has some learning curve, but once you are comfortable with them, you will love them. See one simple solution to your problem. Cheers.
import pandas as pd
import matplotlib.pyplot as plt
from datetime import timedelta
df = pd.read_csv(r"your-folder\dates.csv", parse_dates=[0]) # assuming dates are in the first column
df.sort_values(by="date", ascending=False, inplace=True)
last_date = df["date"].max().date()
start_date = df["date"].max().date() - timedelta(days=365)
dfRange = df[(df["date"].dt.date >= start_date) & (df["date"].dt.date <= last_date)]
plt.scatter(dfRange["txCount"],dfRange["txVolume(USD)"])
plt.xlabel("txCount")
plt.ylabel("txVolume(USD)")
plt.savefig(r"target-location\dates.png")
plt.close()
Answer from griggy on Stack Overflowpython csv to dictionary using csv or pandas module - Stack Overflow
python - Does csv.DicReader() return a dictionary object? - Stack Overflow
python - Load csv file as a dataframe - Stack Overflow
python - How to use csv.DictReader - Stack Overflow
You can do this very simply using pandas.
import pandas as pd
# get only the columns you want from the csv file
df = pd.read_csv(target_path + target_file, usecols=['Column Name1', 'Column Name2'])
result = df.to_dict(orient='records')
Sources:
- pandas.read_csv
- pandas.DataFrame.to_dict
You can use the to_dict method to get a list of dicts:
import pandas as pd
df = pd.read_csv(target_path+target_file, names=Fieldnames)
records = df.to_dict(orient='records')
for row in records:
print row
to_dict documentation:
In [67]: df.to_dict?
Signature: df.to_dict(orient='dict')
Docstring:
Convert DataFrame to dictionary.
Parameters
----------
orient : str {'dict', 'list', 'series', 'split', 'records', 'index'}
Determines the type of the values of the dictionary.
- dict (default) : dict like {column -> {index -> value}}
- list : dict like {column -> [values]}
- series : dict like {column -> Series(values)}
- split : dict like
{index -> [index], columns -> [columns], data -> [values]}
- records : list like
[{column -> value}, ... , {column -> value}]
- index : dict like {index -> {column -> value}}
.. versionadded:: 0.17.0
Abbreviations are allowed. `s` indicates `series` and `sp`
indicates `split`.
Returns
-------
result : dict like {column -> {index -> value}}
File: /usr/local/lib/python2.7/dist-packages/pandas/core/frame.py
Type: instancemethod
csv.reader
Given your desired output, you do not need to use dict or, consequently, csv.DictReader. Instead, just use csv.reader, which returns an iterator. Then use next and list to extract headers and data respectively:
from io import StringIO
import csv
x = StringIO("""play,weather,temperature
yes,sunny,77
no,rainy,60
yes,windy,70""")
# replace x with open('file.csv', 'r')
with x as fin:
reader = csv.reader(fin)
headers = next(reader) # get headers from first row
data = list(reader) # exhaust iterator from second row onwards
The result is a list of headers, and a list of lists for data:
print(headers)
['play', 'weather', 'temperature']
print(data)
[['yes', 'sunny', '77'],
['no', 'rainy', '60'],
['yes', 'windy', '70']]
pandas
If you are willing to use a 3rd party library, Pandas may be a better option as it handles type conversion and indexing more conveniently:
import pandas as pd
df = pd.read_csv('file.csv')
The result is a pd.DataFrame object:
print(df)
play weather temperature
0 yes sunny 77
1 no rainy 60
2 yes windy 70
print(type(df))
<class 'pandas.core.frame.DataFrame'>
DictReader returns a file-like object. It still reads the data from the csv file in one row at a time, but the returned rows are ordered dictionaries instead of lists.
If your files is:
play,weather,temperature
yes,sunny,77
no,rainny,60
yes,windy,70
Then you can use DictReader in the following way:
with open('path/to/file.csv') as fp:
header = fp.readline().strip().split(',')
dreader = DictReader(fp, header)
data = list(dreader)
In this case, data will be a list of OrderedDict objects with the mapping from the headers to each item in the row.
data
#returns:
[OrderedDict([('play', 'yes'), ('weather', 'sunny'), ('temperature', '77')]),
OrderedDict([('play', 'no'), ('weather', 'rainny'), ('temperature', '60')]),
OrderedDict([('play', 'yes'), ('weather', 'windy'), ('temperature', '70')])]
unicodecsv library can also be used to read .csv files.
import unicodecsv
import pandas as pd
def read_csv(data):
""" Returns a list of dicts from .csv file passed in as data"""
with open(data, "rb") as f:
reader = list(unicodecsv.DictReader(f))
return reader
file = read_csv('filename.csv') # call the function and pass in your 'filename'
pd.DataFrame(file) # call pd.DataFrame() function with file as an argument
# to convert it to DataFrame object
import pandas as pd
data = pd.read_csv('file_name')
The csv.DictReader function expects a file object as its first argument, not a string. So, it always is used with normal file io.
for example,
with open('input.csv', 'r') as f:
reader = csv.DictReader(f, delimiter=' ', fieldnames = ['Device', 'tps', 'kB_read/s' ,'kB_wrtn/s', 'kB_dscd/s', 'kB_read', 'kB_wrtn', 'kB_dscd'])
And, you can iterate the reader (i.e., iterable of dicts) row by row.
for row in reader:
print(row)
The row variable contains dictionary-like object that you can easily access by key, for example, row['Device'].
The csv module expects the input to be an iterable containing lines of the CSV. ' '.join(ios_r.stdout.split()) is returning a single string (it simply collapses all the whitespace delimiters into single spaces). Iterating over a string returns loops over the characters, not the lines.
You should perform the join and split on each line, not the entire stdout at once.
ios_e = csv.DictReader(' '.join(line.split()) for line in ios_r.stdout, delimiter = ' ')
You won't need the fieldnames argument in this code, since it will get them from the first line.
Ok, this will make more sense. Why does this work:
import csv
with open('test.csv') as csv_file:
csv_reader = csv.DictReader(csv_file)
for row in csv_reader:
print(row)But this gives an error that the file is closed:
import csv
with open('test.csv') as csv_file:
csv_reader = csv.DictReader(csv_file)
for row in csv_reader:
print(row)
Traceback (most recent call last): File "c:\Users\John\Documents\Python\test_project\test2.py", line 5, in <module> for row in csv_reader: File "C:\Users\John\AppData\Local\Programs\Python\Python311\Lib\csv.py", line 110, in next self.fieldnames File "C:\Users\John\AppData\Local\Programs\Python\Python311\Lib\csv.py", line 97, in fieldnames self._fieldnames = next(self.reader) ^ ValueError: I/O operation on closed file.
Does DictReader (and also reader) not just run once and return a value? Why is csv_file being accessed after the line it's used in?
Thanks!
You could use a library like pandas, it will infer the types for you (it's a bit of an overkill but it does the job).
import pandas
data = pandas.read_csv(r'..\data\data.csv')
# if you just want to retrieve the first column as a list of int do
list(data.Col1)
>>> [1, 90]
# to convert the whole CSV file to a list of dict use
data.transpose().to_dict().values()
>>> [{' Col2': 2, ' Col3': 3, 'Col1': 1}, {' Col2': 2, ' Col3': 3, 'Col1': 90}]
Alternatively here is an implementation of a typed DictReader:
from csv import DictReader
from itertools import imap, izip
class TypedDictReader(DictReader):
def __init__(self, f, fieldnames=None, restkey=None, restval=None, \
dialect="excel", fieldtypes=None, *args, **kwds):
DictReader.__init__(self, f, fieldnames, restkey, restval, dialect, *args, **kwds)
self._fieldtypes = fieldtypes
def next(self):
d = DictReader.next(self)
if len(self._fieldtypes) >= len(d) :
# extract the values in the same order as the csv header
ivalues = imap(d.get, self._fieldnames)
# apply type conversions
iconverted = (x(y) for (x,y) in izip(self._fieldtypes, ivalues))
# pass the field names and the converted values to the dict constructor
d = dict(izip(self._fieldnames, iconverted))
return d
and here is how to use it:
reader = TypedDictReader(open('..\data\data.csv'), dialect='excel', \
fieldtypes=[int, int, int])
list(reader)
>>> [{' Col2': 2, ' Col3': 3, 'Col1': 1}, {' Col2': 2, ' Col3': 3, 'Col1': 90}]
If you quote the non-numeric values in the csv file and initialize the reader by
pol = csv.DictReader(open('..\data\data.csv'),
quoting=csv.QUOTE_NONNUMERIC, dialect="excel")
then numeric values will be automatically converted to floats.