Python 3:
The official csv documentation recommends opening the file with newline='' on all platforms to disable universal newlines translation:
with open('output.csv', 'w', newline='', encoding='utf-8') as f:
writer = csv.writer(f)
...
The CSV writer terminates each line with the lineterminator of the dialect, which is '\r\n' for the default excel dialect on all platforms because that's what RFC 4180 recommends.
Python 2:
On Windows, always open your files in binary mode ("rb" or "wb"), before passing them to csv.reader or csv.writer.
Although the file is a text file, CSV is regarded a binary format by the libraries involved, with \r\n separating records. If that separator is written in text mode, the Python runtime replaces the \n with \r\n, hence the \r\r\n observed in the file.
See this previous answer.
Answer from John Machin on Stack OverflowPython 3:
The official csv documentation recommends opening the file with newline='' on all platforms to disable universal newlines translation:
with open('output.csv', 'w', newline='', encoding='utf-8') as f:
writer = csv.writer(f)
...
The CSV writer terminates each line with the lineterminator of the dialect, which is '\r\n' for the default excel dialect on all platforms because that's what RFC 4180 recommends.
Python 2:
On Windows, always open your files in binary mode ("rb" or "wb"), before passing them to csv.reader or csv.writer.
Although the file is a text file, CSV is regarded a binary format by the libraries involved, with \r\n separating records. If that separator is written in text mode, the Python runtime replaces the \n with \r\n, hence the \r\r\n observed in the file.
See this previous answer.
While @john-machin gives a good answer, it's not always the best approach. For example, it doesn't work on Python 3 unless you encode all of your inputs to the CSV writer. Also, it doesn't address the issue if the script wants to use sys.stdout as the stream.
I suggest instead setting the 'lineterminator' attribute when creating the writer:
import csv
import sys
doc = csv.writer(sys.stdout, lineterminator='\n')
doc.writerow('abc')
doc.writerow(range(3))
That example will work on Python 2 and Python 3 and won't produce the unwanted newline characters. Note, however, that it may produce undesirable newlines (omitting the LF character on Unix operating systems).
In most cases, however, I believe that behavior is preferable and more natural than treating all CSV as a binary format. I provide this answer as an alternative for your consideration.
So, basically, I've got this code:
new_list_csv = []
def add_it():
#gets the values from Entry
#three different tk Entrys generate three different values
name= self.e.get()
ex= self.e_x.get()
ey= self.e_y.get()
#adds them to the new list
new_list_csv.append(name)
new_list_csv.append(ex)
new_list_csv.append(ey)
#appends new characters to csv file
with open ('chr_list_copy.txt', 'a', newline='') as write_obj:
csv_writer = csv.writer(write_obj)
csv_writer.writerow(new_list_csv)and the csv file looks like this:
name,x,y Ergo,1,1 Sum,5,5 Name12,1,2 Name34,3,4 Name56,5,6 Name78,7,8 Name910,9,10 Name1112,11,12 Name1314,13,14
if I try and add [Otto,6,9], I get:
name,x,y [...] Name1314,13,14Otto,6,9
instead of:
name,x,y [...] Name1314,13,14 Otto,6,9
I've used this very same structure for the rest of my code, but in this specific instance, it doesn't work.
I am running the script on a Windows 7 machine using Python 3.4.1. The script runs correctly and produces a csv file, the only problem is at the end of each line is a '\r\n' which causes an extra blank line to appear when displayed in Excel. How do get remove the extra blank line?
import pyodbc
import csv
connect = pyodbc.connect('driver={SQL Server Native Client 10.0};SERVER=xxxxx-7- VM;DATABASE=DEMO_xxxx_TEST;UID=DEMO_xxxx_TEST;PWD=password')
cursor = connect.cursor()
cursor.execute('''select Report_name
,UserID
,FieldName
,XPos
,YPos
,Hidden
,Picture
from Rpt_Nudge_2
where Report_Name = 'UB04' and UserID = 'myid' and YPos > 8886 and YPos < 9888
order by YPos, XPos''')
col_names = [i[0] for i in cursor.description]
print(col_names)
nudge = cursor.fetchall()
with open('UB04_nudge.csv', 'w') as csvfile:
fileout = csv.writer(csvfile)
row = fileout.writerow(col_names)
for line in nudge:
fileout.writerow(line)
connect.close()
The strip() method removes whitespace, including newlines.
fileout.writerow(line.strip())
In Python 2, you could write to CSV files with the 'wb' option on the file and avoid this.
In Python 3, it's a little different - here's the documentation, take a look at the footnote.
Since you're opening the csv file as a file, you should replace line 25 with this:
with open('UB04_nudge.csv', 'w', newline='') as csvfile:
Basically, since Windows uses \r\n line endings, file() is already planning to write a newline ending out after each line. CSV does this as well - so you're getting the duplicate newlines after each row. By setting newline='', you're telling file() to not terminate new lines - which works since csv() will terminate the lines on its own.
The csv.writer module directly controls line endings and writes \r\n into the file directly. In Python 3 the file must be opened in untranslated text mode with the parameters 'w', newline='' (empty string) or it will write \r\r\n on Windows, where the default text mode will translate each \n into \r\n.
#!python3
with open('/pythonwork/thefile_subset11.csv', 'w', newline='') as outfile:
writer = csv.writer(outfile)
If using the Path module:
from pathlib import Path
import csv
with Path('/pythonwork/thefile_subset11.csv').open('w', newline='') as outfile:
writer = csv.writer(outfile)
If using the StringIO module to build an in-memory result, the result string will contain the translated line terminator:
from io import StringIO
import csv
s = StringIO()
writer = csv.writer(s)
writer.writerow([1,2,3])
print(repr(s.getvalue())) # '1,2,3\r\n' (Windows result)
If writing that string to a file later, remember to use newline='':
# built-in open()
with open('/pythonwork/thefile_subset11.csv', 'w', newline='') as f:
f.write(s.getvalue())
# Path's open()
with Path('/pythonwork/thefile_subset11.csv').open('w', newline='') as f:
f.write(s.getvalue())
# Path's write_text() added the newline parameter to Python 3.10.
Path('/pythonwork/thefile_subset11.csv').write_text(s.getvalue(), newline='')
In Python 2, use binary mode to open outfile with mode 'wb' instead of 'w' to prevent Windows newline translation. Python 2 also has problems with Unicode and requires other workarounds to write non-ASCII text. See the Python 2 link below and the UnicodeReader and UnicodeWriter examples at the end of the page if you have to deal with writing Unicode strings to CSVs on Python 2, or look into the 3rd party unicodecsv module:
#!python2
with open('/pythonwork/thefile_subset11.csv', 'wb') as outfile:
writer = csv.writer(outfile)
Documentation Links
- https://docs.python.org/3/library/csv.html#csv.writer
- https://docs.python.org/2/library/csv.html#csv.writer
Opening the file in binary mode "wb" will not work in Python 3+. Or rather, you'd have to convert your data to binary before writing it. That's just a hassle.
Instead, you should keep it in text mode, but override the newline as empty. Like so:
with open('/pythonwork/thefile_subset11.csv', 'w', newline='') as outfile:
Use file.seek to move file pointer before the last \r\n, then use file.truncate.
import os
import csv
with open('eggs.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
spamwriter.writerow(['Spam'] * 5 + ['Baked Beans'])
spamwriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
csvfile.seek(-2, os.SEEK_END) # <---- 2 : len('\r\n')
csvfile.truncate() # <----
NOTE: You should change -2 if you use different lineterminator. I used -2 because \r\n is default lineterminator.
here is a solution that removes the newline symbols from last line of csv, using rstrip:
def remove_last_line_from_csv(filename):
with open(filename) as myFile:
lines = myFile.readlines()
last_line = lines[len(lines)-1]
lines[len(lines)-1] = last_line.rstrip()
with open(filename, 'w') as myFile:
myFile.writelines(lines)
Here is a simple solution: Replace all \n with \\n before saving to CSV. This will preserve the newline characters.
df.loc[:, "Column_Name"] = df["Column_Name"].apply(lambda x: x.replace('\n', '\\n'))
df.to_csv("df.csv", index=False)
I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, although the principle could be applied to Python 2.
The main trick
This is to replace the \n characters before writing with a weird character that otherwise wouldn't be included, then to swap that weird character back for \n after reading the file back from disk.
For my weird character, I will use the Icelandic thorn: Þ, but you can choose anything that should otherwise not appear in your text variables. Its name, as defined in the standardised Unicode specification is: LATIN SMALL LETTER THORN. You can use it in Python 3 a couple of ways:
weird_literal = 'þ'
weird_name = '\N{LATIN SMALL LETTER THORN}'
weird_char = '\xfe' # hex representation
weird_literal == weird_name == weird_char # True
That \N is pretty cool (and works in python 3.6 inside formatted strings too)... it basically allows you to pass the Name of a character, as per Unicode's specification.
An alternative character that may serve as a good standard is '\u2063' (INVISIBLE SEPARATOR).
Replacing \n
Now we use this weird character to replace '\n'. Here are the two ways that pop into my mind for achieving this:
using a list comprehension on your list of lists:
data:new_data = [[sample[0].replace('\n', weird_char) + weird_char, sample[1]] for sample in data]putting the data into a dataframe, and using replace on the whole
textcolumn in one godf1 = pd.DataFrame(data, columns=['text', 'category']) df1.text = df.text.str.replace('\n', weird_char)
The resulting dataframe looks like this, with newlines replaced:
text category
0 some text in one line 1
1 text withþnew line character 0
2 another newþline character 1
Writing the results to disk
Now we write either of those identical dataframes to disk. I set index=False as you said you don't want row numbers to be in the CSV:
FILE = '~/path/to/test_file.csv'
df.to_csv(FILE, index=False)
What does it look like on disk?
text,category
some text in one line,1
text withþnew line character,0
another newþline character,1
Getting the original data back from disk
Read the data back from file:
new_df = pd.read_csv(FILE)
And we can replace the Þ characters back to \n:
new_df.text = new_df.text.str.replace(weird_char, '\n')
And the final DataFrame:
new_df
text category
0 some text in one line 1
1 text with\nnew line character 0
2 another new\nline character 1
If you want things back into your list of lists, then you can do this:
original_lists = [[text, category] for index, text, category in old_df_again.itertuples()]
Which looks like this:
[['some text in one line', 1],
['text with\nnew line character', 0],
['another new\nline character', 1]]
This problem occurs only with Python on Windows.
In Python v3, you need to add newline='' in the open call per:
Python 3.3 CSV.Writer writes extra blank rows
On Python v2, you need to open the file as binary with "b" in your open() call before passing to csv
Changing the line
with open('stocks2.csv','w') as f:
to:
with open('stocks2.csv','wb') as f:
will fix the problem
More info about the issue here:
CSV in Python adding an extra carriage return, on Windows
I came across this issue on windows for Python 3. I tried changing newline parameter while opening file and it worked properly with newline=''.
Add newline='' to open() method as follows:
with open('stocks2.csv','w', newline='') as f:
f_csv = csv.DictWriter(f, headers)
f_csv.writeheader()
f_csv.writerows(rows)
It will work as charm.
Hope it helps.
Recommended implementation per Python3 Documentation.
with open('records.csv','w', newline='') as csvfile:
#creating a csv writer object
csvwriter = csv.writer(csvfile)
#writing the fields
csvwriter.writerow(fields)
# writing the data rows
csvwriter.writerows(rows)
https://docs.python.org/3/library/csv.html#csv.writer
Method 1 So when ever i write csv files spaces between lines are created and when reading the files they create problems for me So here is how i solved it
with open("Location.csv","r") as obj:
reader=csv.reader(obj)
for lines in reader:
try:
print(lines["Code"])
print(lines["Key_num"])
except TypeError:
pass
Method 2 Or even simpler you can use Dictreader works fine without error even if spaces are preset
with open("Location.csv","r") as obj:
reader=csv.DictReader(obj)
for lines in reader:
print(lines["Code"])