Here are some minimal complete examples how to read CSV files and how to write CSV files with Python.
Pure Python:
import csv
# Define data
data = [
(1, "A towel,", 1.0),
(42, " it says, ", 2.0),
(1337, "is about the most ", -1),
(0, "massively useful thing ", 123),
(-2, "an interstellar hitchhiker can have.", 3),
]
# Write CSV file
with open("test.csv", "wt") as fp:
writer = csv.writer(fp, delimiter=",")
# writer.writerow(["your", "header", "foo"]) # write header
writer.writerows(data)
# Read CSV file
with open("test.csv") as fp:
reader = csv.reader(fp, delimiter=",", quotechar='"')
# next(reader, None) # skip the headers
data_read = [row for row in reader]
print(data_read)
After that, the contents of data_read are
[['1', 'A towel,', '1.0'],
['42', ' it says, ', '2.0'],
['1337', 'is about the most ', '-1'],
['0', 'massively useful thing ', '123'],
['-2', 'an interstellar hitchhiker can have.', '3']]
Please note that CSV reads only strings. You need to convert to the column types manually.
A Python 2+3 version was here before (link), but Python 2 support is dropped. Removing the Python 2 stuff massively simplified this answer.
Related
- How do I write data into csv format as string (not file)?
- How can I use io.StringIO() with the csv module?: This is interesting if you want to serve a CSV on-the-fly with Flask, without actually storing the CSV on the server.
mpu
Have a look at my utility package mpu for a super simple and easy to remember one:
import mpu.io
data = mpu.io.read('example.csv', delimiter=',', quotechar='"', skiprows=None)
mpu.io.write('example.csv', data)
Pandas
import pandas as pd
# Read the CSV into a pandas data frame (df)
# With a df you can do many things
# most important: visualize data with Seaborn
df = pd.read_csv('myfile.csv', sep=',')
print(df)
# Or export it in many ways, e.g. a list of tuples
tuples = [tuple(x) for x in df.values]
# or export it as a list of dicts
dicts = df.to_dict().values()
See read_csv docs for more information. Please note that pandas automatically infers if there is a header line, but you can set it manually, too.
If you haven't heard of Seaborn, I recommend having a look at it.
Other
Reading CSV files is supported by a bunch of other libraries, for example:
dask.dataframe.read_csvspark.read.csv
Created CSV file
1,"A towel,",1.0
42," it says, ",2.0
1337,is about the most ,-1
0,massively useful thing ,123
-2,an interstellar hitchhiker can have.,3
Common file endings
.csv
Working with the data
After reading the CSV file to a list of tuples / dicts or a Pandas dataframe, it is simply working with this kind of data. Nothing CSV specific.
Alternatives
- JSON: Nice for writing human-readable data; VERY commonly used (read & write)
- CSV: Super simple format (read & write)
- YAML: Nice to read, similar to JSON (read & write)
- pickle: A Python serialization format (read & write)
- MessagePack (Python package): More compact representation (read & write)
- HDF5 (Python package): Nice for matrices (read & write)
- XML: exists too *sigh* (read & write)
For your application, the following might be important:
- Support by other programming languages
- Reading / writing performance
- Compactness (file size)
See also: Comparison of data serialization formats
In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python
Answer from Martin Thoma on Stack Overflowpython - How do I read and write CSV files? - Stack Overflow
python - How do I make a proper csv file using DictWriter method - Stack Overflow
CSV writer help
Python: Writing a Dict to CSV - Stack Overflow
Here are some minimal complete examples how to read CSV files and how to write CSV files with Python.
Pure Python:
import csv
# Define data
data = [
(1, "A towel,", 1.0),
(42, " it says, ", 2.0),
(1337, "is about the most ", -1),
(0, "massively useful thing ", 123),
(-2, "an interstellar hitchhiker can have.", 3),
]
# Write CSV file
with open("test.csv", "wt") as fp:
writer = csv.writer(fp, delimiter=",")
# writer.writerow(["your", "header", "foo"]) # write header
writer.writerows(data)
# Read CSV file
with open("test.csv") as fp:
reader = csv.reader(fp, delimiter=",", quotechar='"')
# next(reader, None) # skip the headers
data_read = [row for row in reader]
print(data_read)
After that, the contents of data_read are
[['1', 'A towel,', '1.0'],
['42', ' it says, ', '2.0'],
['1337', 'is about the most ', '-1'],
['0', 'massively useful thing ', '123'],
['-2', 'an interstellar hitchhiker can have.', '3']]
Please note that CSV reads only strings. You need to convert to the column types manually.
A Python 2+3 version was here before (link), but Python 2 support is dropped. Removing the Python 2 stuff massively simplified this answer.
Related
- How do I write data into csv format as string (not file)?
- How can I use io.StringIO() with the csv module?: This is interesting if you want to serve a CSV on-the-fly with Flask, without actually storing the CSV on the server.
mpu
Have a look at my utility package mpu for a super simple and easy to remember one:
import mpu.io
data = mpu.io.read('example.csv', delimiter=',', quotechar='"', skiprows=None)
mpu.io.write('example.csv', data)
Pandas
import pandas as pd
# Read the CSV into a pandas data frame (df)
# With a df you can do many things
# most important: visualize data with Seaborn
df = pd.read_csv('myfile.csv', sep=',')
print(df)
# Or export it in many ways, e.g. a list of tuples
tuples = [tuple(x) for x in df.values]
# or export it as a list of dicts
dicts = df.to_dict().values()
See read_csv docs for more information. Please note that pandas automatically infers if there is a header line, but you can set it manually, too.
If you haven't heard of Seaborn, I recommend having a look at it.
Other
Reading CSV files is supported by a bunch of other libraries, for example:
dask.dataframe.read_csvspark.read.csv
Created CSV file
1,"A towel,",1.0
42," it says, ",2.0
1337,is about the most ,-1
0,massively useful thing ,123
-2,an interstellar hitchhiker can have.,3
Common file endings
.csv
Working with the data
After reading the CSV file to a list of tuples / dicts or a Pandas dataframe, it is simply working with this kind of data. Nothing CSV specific.
Alternatives
- JSON: Nice for writing human-readable data; VERY commonly used (read & write)
- CSV: Super simple format (read & write)
- YAML: Nice to read, similar to JSON (read & write)
- pickle: A Python serialization format (read & write)
- MessagePack (Python package): More compact representation (read & write)
- HDF5 (Python package): Nice for matrices (read & write)
- XML: exists too *sigh* (read & write)
For your application, the following might be important:
- Support by other programming languages
- Reading / writing performance
- Compactness (file size)
See also: Comparison of data serialization formats
In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python
If you are working with CSV data and want a solution with a smaller footprint than pandas, you can try my package, littletable. Can be pip-installed, or just dropped in as a single .py file with your own code, so very portable and suitable for serverless apps.
Reading CSV data is as simple as calling csv_import:
data = """\
1,"A towel,",1.0
42," it says, ",2.0
1337,is about the most ,-1
0,massively useful thing ,123
-2,an interstellar hitchhiker can have.,3"""
import littletable as lt
tbl = lt.Table().csv_import(data, fieldnames="number1,words,number2".split(','))
tbl.present()
Prints:
Number1 Words Number2
──────────────────────────────────────────────────────────
1 A towel, 1.0
42 it says, 2.0
1337 is about the most -1
0 massively useful thing 123
-2 an interstellar hitchhiker can have. 3
(littletable uses the rich module for presenting Tables.)
littletable doesn't automatically try to convert numeric data, so a numeric transform function is needed for the numeric columns.
def get_numeric(s):
try:
return int(s)
except ValueError:
try:
return float(s)
except ValueError:
return s
tbl = lt.Table().csv_import(
data,
fieldnames="number1,words,number2".split(','),
transforms={}.fromkeys("number1 number2".split(), get_numeric)
)
tbl.present()
This gives:
Number1 Words Number2
──────────────────────────────────────────────────────────
1 A towel, 1.0
42 it says, 2.0
1337 is about the most -1
0 massively useful thing 123
-2 an interstellar hitchhiker can have. 3
The numeric columns are right-justified instead of left-justified.
littletable also has other ORM-ish features, such as indexing, joining, pivoting, and full-text search. Here is a table of statistics on the numeric columns:
tbl.stats("number1 number2".split()).present()
Name Mean Min Max Variance Std_Dev Count Missing
────────────────────────────────────────────────────────────────────────────────
number1 275.6 -2 1337 352390.3 593.6247130974249 5 0
number2 25.6 -1 123 2966.8 54.468339427597755 5 0
or transposed:
tbl.stats("number1 number2".split(), by_field=False).present()
Stat Number1 Number2
───────────────────────────────────────────────────
mean 275.6 25.6
min -2 -1
max 1337 123
variance 352390.3 2966.8
std_dev 593.6247130974249 54.468339427597755
count 5 5
missing 0 0
Other formats can be output too, such as Markdown:
print(tbl.stats("number1 number2".split(), by_field=False).as_markdown())
| stat | number1 | number2 |
|---|---:|---:|
| mean | 275.6 | 25.6 |
| min | -2 | -1 |
| max | 1337 | 123 |
| variance | 352390.3 | 2966.8 |
| std_dev | 593.6247130974249 | 54.468339427597755 |
| count | 5 | 5 |
| missing | 0 | 0 |
Which would render from Markdown as
| stat | number1 | number2 |
|---|---|---|
| mean | 275.6 | 25.6 |
| min | -2 | -1 |
| max | 1337 | 123 |
| variance | 352390.3 | 2966.8 |
| std_dev | 593.6247130974249 | 54.468339427597755 |
| count | 5 | 5 |
| missing | 0 | 0 |
Lastly, here is a text search on the words for any entry with the word "hitchhiker":
tbl.create_search_index("words")
for match in tbl.search.words("hitchhiker"):
print(match)
Prints:
namespace(number1=-2, words='an interstellar hitchhiker can have.', number2=3)
Since you have all lists, it would be easier to use csv.writer instead of Dictwriter:
In [182]: with open('test.csv', 'w', newline='') as csvfile:
...: writer = csv.writer(csvfile)
...: writer.writerow(types)
...: writer.writerows(zip(fruits, vegetables, spices))
...:
Output:
Fruits,Vegetables,Spices
apple,onion,salt
banana,garlic,pepper
grape,carrot,paprika
orange,cabbage,cumin
Use pandas:
import pandas as pd
fruits = ["apple", "banana", "grape", "orange"]
vegetables = ["onion", "garlic", "carrot", "cabbage"]
spices = ["salt", "pepper", "paprika", "cumin"]
types = ["Fruits", "Vegetables", "Spices"]
df = pd.DataFrame( list(zip(fruits,vegetables,spices)), columns = types)
df.to_csv(r'file_name.csv', index = False)
Result:
Fruits,Vegetables,Spices
apple,onion,salt
banana,garlic,pepper
grape,carrot,paprika
orange,cabbage,cumin
for row in r:
for val in row:
if len(row[val])<1:
row[val]='0'
w.write_row(row)
you have to actually tell it to write the row
If you really want to use the csv module there are multiple errors in your code you need to fix, you need to pass the fieldnames to csv.DictWriter then write them and the length of the empty string will be 1 so csv.DictWriter will never be True so nothing will be changed:
with open('in.csv', 'r') as infile, open('Fixed.csv', 'w') as outfile:
r = csv.DictReader(infile)
w = csv.DictWriter(outfile, fieldnames=r.fieldnames)
w.writeheader()
for row in r:
for k, v in row.items():
if v == "''":
row[k] = "0"
w.writerow(row)
To change the original file it would actually be easier just use str.replace and open the file without the csv module, to change the original file you can use fileinput with inplace=True:
import fileinput
import sys
for line in fileinput.input("in.csv",inplace=True):
sys.stdout.write(line.replace("''","0"))
Output:
One,Two,Three,Four
1,0,0,1
0,0,1,0
1,0,1,0
0,0,0,1
Or using a NamedTemporaryFile with shutil.move, writing to the tempfile the replacing the original file with the updated file using move:
from tempfile import NamedTemporaryFile
from shutil import move
with open('TestCSV.csv', 'r') as infile, NamedTemporaryFile(dir=".", delete=False) as outfile:
for line in infile:
outfile.write(line.replace("''", "0"))
move(outfile.name,'TestCSV.csv')
Or if you want a new file just do the same using file.write:
with open('TestCSV.csv','r') as infile,open('Fixed.csv','w') as outfile:
for line in infile:
outfile.write(line.replace("''","0"))
You only have either an empty string or a "1" in each line so it is simpler just to replace the empty strings.
You could also do it quite easily with pandas using a Dataframe and df.replace if you intend to actually create a df with your data:
df = pd.read_csv("in.csv")
df.replace("''","0",inplace=True)
print(df)
One Two Three Four
0 1 0 0 1
1 0 0 1 0
2 1 0 1 0
3 0 0 0 1
# work on df
.......
# save results to csv
df.to_csv("fixed.csv",index=False)
Output:
One,Two,Three,Four
1,0,0,1
0,0,1,0
1,0,1,0
0,0,0,1