Note that, as the docs say:
csvfile can be any object which supports the iterator protocol and returns a string each time its
next()method is called — file objects and list objects are both suitable.
So, you can always stick a filter on the file before handing it to your reader or DictReader. Instead of this:
with open('myfile.csv', 'rU') as myfile:
for row in csv.reader(myfile):
Do this:
with open('myfile.csv', 'rU') as myfile:
filtered = (line.replace('\r', '') for line in myfile)
for row in csv.reader(filtered):
That '\r' is the Python (and C) way of spelling ^M. So, this just strips all ^M characters out, no matter where they appear, by replacing each one with an empty string.
I guess I want to modify the file permanently as opposed to filtering it.
First, if you want to modify the file before running your Python script on it, why not do that from outside of Python? sed, tr, many text editors, etc. can all do this for you. Here's a GNU sed example:
gsed -i'' 's/\r//g' myfile.csv
But if you want to do it in Python, it's not that much more verbose, and you might find it more readable, so:
First, you can't really modify a file in-place if you want to insert or delete from the middle. The usual solution is to write a new file, and either move the new file over the old one (Unix only) or delete the old one (cross-platform).
The cross-platform version:
os.rename('myfile.csv', 'myfile.csv.bak')
with open('myfile.csv.bak', 'rU') as infile, open('myfile.csv', 'wU') as outfile:
for line in infile:
outfile.write(line.replace('\r'))
os.remove('myfile.csv.bak')
The less-clunky, but Unix-only, version:
temp = tempfile.NamedTemporaryFile(delete=False)
with open('myfile.csv', 'rU') as myfile, closing(temp):
for line in myfile:
temp.write(line.replace('\r'))
os.rename(tempfile.name, 'myfile.csv')
Answer from abarnert on Stack OverflowI am running the script on a Windows 7 machine using Python 3.4.1. The script runs correctly and produces a csv file, the only problem is at the end of each line is a '\r\n' which causes an extra blank line to appear when displayed in Excel. How do get remove the extra blank line?
import pyodbc
import csv
connect = pyodbc.connect('driver={SQL Server Native Client 10.0};SERVER=xxxxx-7- VM;DATABASE=DEMO_xxxx_TEST;UID=DEMO_xxxx_TEST;PWD=password')
cursor = connect.cursor()
cursor.execute('''select Report_name
,UserID
,FieldName
,XPos
,YPos
,Hidden
,Picture
from Rpt_Nudge_2
where Report_Name = 'UB04' and UserID = 'myid' and YPos > 8886 and YPos < 9888
order by YPos, XPos''')
col_names = [i[0] for i in cursor.description]
print(col_names)
nudge = cursor.fetchall()
with open('UB04_nudge.csv', 'w') as csvfile:
fileout = csv.writer(csvfile)
row = fileout.writerow(col_names)
for line in nudge:
fileout.writerow(line)
connect.close()
The strip() method removes whitespace, including newlines.
fileout.writerow(line.strip())
In Python 2, you could write to CSV files with the 'wb' option on the file and avoid this.
In Python 3, it's a little different - here's the documentation, take a look at the footnote.
Since you're opening the csv file as a file, you should replace line 25 with this:
with open('UB04_nudge.csv', 'w', newline='') as csvfile:
Basically, since Windows uses \r\n line endings, file() is already planning to write a newline ending out after each line. CSV does this as well - so you're getting the duplicate newlines after each row. By setting newline='', you're telling file() to not terminate new lines - which works since csv() will terminate the lines on its own.
I'm processing a file with pandas and saving it to CSV. For some reason the newly produced CSV file has a new line character at the end of it. All of my googling shows how to deal with all newlines or it's more specific than I need.
What is the easiest way to remove the final newline character in a CSV file?
The same question was asked earlier today here. Please check my comments on that regarding whether that would actually be a problem and the resolution if you really must resolve it.
I'm on mobile, but everything you need to know is on this page:
https://stackoverflow.com/questions/18857352/python-remove-very-last-character-in-file
The second answer handles utf-8 encoding beautifully, but it specifically skips newlines and you want to delete the last newline. You'll have to rejigger the logic accordingly.
Using Miller (mlr), a CSV-aware multi-purpose processing utility for various structured document formats, to clean up the whitespace of all fields:
$ cat file
"host1","host1","linux
server",""
"host2","host2","linux server",""
$ mlr --csv -N clean-whitespace file
host1,host1,linux server,
host2,host2,linux server,
This reads the data in file as header-less CSV records and applies the clean-whitespace operation to each. The clean-whitespace operation trims flanking whitespace from each field's value and combines consecutive whitespace characters into single spaces.
To instead only replace newlines with spaces, you may iterate over the fields with a short put expression:
*) { $[k] = gssub(v, "\n", " ") }' file
host1,host1,linux server,
host2,host2,linux server,
The gssub() function acts like gsub() in Awk, but does not treat its query argument like a regular expression (Miller also has gsub()).
If you feel you need to have the fields quoted even though it's not strictly needed (Miller adds quotes automatically if a field's value requires it), then use mlr with its --quote-all option:
$ mlr --csv -N --quote-all clean-whitespace file
"host1","host1","linux server",""
"host2","host2","linux server",""
$ mlr --csv -N --quote-all put 'for (k,v in $*) { $[k] = gssub(v, "\n", " ") }' file
"host1","host1","linux server",""
"host2","host2","linux server",""
The last thing you want to do is try to do this in bash. See Why is using a shell loop to process text considered bad practice?.
Now, if what you want can be expressed as "remove any newline characters unless they come right after a " character", you could do something like this:
perl -pe 's/(?<!")\n/ /g' file
The (?<!")\n matches any newline character that is NOT preceded by a ". So given an example input like this:
$ cat file
"host0","host0","linux
server",""
"host1","host1","linux
server
centos",""
"host2","host2","linux server",""
The command above gives:
$ perl -pe 's/(?<!")\n/ /g' file
"host0","host0","linux server",""
"host1","host1","linux server centos",""
"host2","host2","linux server",""
But, really, mlr is the best approach.
For a uni assignment, I need to process a csv file. However these csv files were written such that the EOL character in every case is a carriage return (\r) character. The assignment prompt says that I cannot read in the file using the csv module as it will not recognise the \r character and thus the requirement of the assignment is that, in my Python script, I change the carriage return to a newline or any other EOL character that Python's csv module can recognise as a valid EOL. How would I do this? Also why doesn't the csv module recognise the carriage return? (Sorry for poor English and/or bad formatting)