Note that, as the docs say:

csvfile can be any object which supports the iterator protocol and returns a string each time its next() method is called — file objects and list objects are both suitable.

So, you can always stick a filter on the file before handing it to your reader or DictReader. Instead of this:

with open('myfile.csv', 'rU') as myfile:
    for row in csv.reader(myfile):

Do this:

with open('myfile.csv', 'rU') as myfile:
    filtered = (line.replace('\r', '') for line in myfile)
    for row in csv.reader(filtered):

That '\r' is the Python (and C) way of spelling ^M. So, this just strips all ^M characters out, no matter where they appear, by replacing each one with an empty string.


I guess I want to modify the file permanently as opposed to filtering it.

First, if you want to modify the file before running your Python script on it, why not do that from outside of Python? sed, tr, many text editors, etc. can all do this for you. Here's a GNU sed example:

gsed -i'' 's/\r//g' myfile.csv

But if you want to do it in Python, it's not that much more verbose, and you might find it more readable, so:

First, you can't really modify a file in-place if you want to insert or delete from the middle. The usual solution is to write a new file, and either move the new file over the old one (Unix only) or delete the old one (cross-platform).

The cross-platform version:

os.rename('myfile.csv', 'myfile.csv.bak')
with open('myfile.csv.bak', 'rU') as infile, open('myfile.csv', 'wU') as outfile:
    for line in infile:
        outfile.write(line.replace('\r'))
os.remove('myfile.csv.bak')

The less-clunky, but Unix-only, version:

temp = tempfile.NamedTemporaryFile(delete=False)
with open('myfile.csv', 'rU') as myfile, closing(temp):
    for line in myfile:
        temp.write(line.replace('\r'))
os.rename(tempfile.name, 'myfile.csv')
Answer from abarnert on Stack Overflow
Top answer
1 of 1
18

Note that, as the docs say:

csvfile can be any object which supports the iterator protocol and returns a string each time its next() method is called — file objects and list objects are both suitable.

So, you can always stick a filter on the file before handing it to your reader or DictReader. Instead of this:

with open('myfile.csv', 'rU') as myfile:
    for row in csv.reader(myfile):

Do this:

with open('myfile.csv', 'rU') as myfile:
    filtered = (line.replace('\r', '') for line in myfile)
    for row in csv.reader(filtered):

That '\r' is the Python (and C) way of spelling ^M. So, this just strips all ^M characters out, no matter where they appear, by replacing each one with an empty string.


I guess I want to modify the file permanently as opposed to filtering it.

First, if you want to modify the file before running your Python script on it, why not do that from outside of Python? sed, tr, many text editors, etc. can all do this for you. Here's a GNU sed example:

gsed -i'' 's/\r//g' myfile.csv

But if you want to do it in Python, it's not that much more verbose, and you might find it more readable, so:

First, you can't really modify a file in-place if you want to insert or delete from the middle. The usual solution is to write a new file, and either move the new file over the old one (Unix only) or delete the old one (cross-platform).

The cross-platform version:

os.rename('myfile.csv', 'myfile.csv.bak')
with open('myfile.csv.bak', 'rU') as infile, open('myfile.csv', 'wU') as outfile:
    for line in infile:
        outfile.write(line.replace('\r'))
os.remove('myfile.csv.bak')

The less-clunky, but Unix-only, version:

temp = tempfile.NamedTemporaryFile(delete=False)
with open('myfile.csv', 'rU') as myfile, closing(temp):
    for line in myfile:
        temp.write(line.replace('\r'))
os.rename(tempfile.name, 'myfile.csv')
🌐
Reddit
reddit.com › r/learnpython › how to remove '\r\n' from the csv records being written?
r/learnpython on Reddit: How to remove '\r\n' from the csv records being written?
July 30, 2015 -

I am running the script on a Windows 7 machine using Python 3.4.1. The script runs correctly and produces a csv file, the only problem is at the end of each line is a '\r\n' which causes an extra blank line to appear when displayed in Excel. How do get remove the extra blank line?

import pyodbc
import csv

connect = pyodbc.connect('driver={SQL Server Native       Client 10.0};SERVER=xxxxx-7-   VM;DATABASE=DEMO_xxxx_TEST;UID=DEMO_xxxx_TEST;PWD=password')

cursor = connect.cursor()

cursor.execute('''select Report_name
                    ,UserID
                    ,FieldName
                    ,XPos
                    ,YPos
                    ,Hidden
                    ,Picture
              from  Rpt_Nudge_2
              where Report_Name = 'UB04' and UserID = 'myid' and YPos > 8886 and YPos < 9888
              order by YPos, XPos''')

col_names = [i[0] for i in cursor.description]
print(col_names)

nudge = cursor.fetchall()

with open('UB04_nudge.csv', 'w') as csvfile:
    fileout = csv.writer(csvfile)
    row = fileout.writerow(col_names)
    for line in nudge:
        fileout.writerow(line)

connect.close()
🌐
Gitlab
tayloramurphy.gitlab.io › removing-newlines-csv
Removing newlines from CSV
January 25, 2016 - The actual Alfred script runs python csvnewlineremove.py {query} in bash. ... # -*- coding: utf-8 -*- import csv, os, sys import StringIO input_file = sys.argv[1] with open(input_file, "rb") as csv_file: csv_data = StringIO.StringIO(csv_file.read()) final_location = os.path.expanduser("~/Desktop/alfredfixed.csv") with open(final_location, "wb") as output: mywriter = csv.writer(output) filtered = (line.replace('\r', '') for line in csv_data) for record in csv.reader(filtered): mywriter.writerow(tuple(record))
🌐
Python
docs.python.org › 3 › library › csv.html
csv — CSV File Reading and Writing
Any other optional or keyword arguments are passed to the underlying writer instance. Note that unlike the DictReader class, the fieldnames parameter of the DictWriter class is not optional. If the argument passed to fieldnames is an iterator, it will be coerced to a list. ... import csv with open('names.csv', 'w', newline='') as csvfile: fieldnames = ['first_name', 'last_name'] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader() writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'}) writer.writerow({'first_name': 'Lovely', 'last_name': 'Spam'}) writer.writerow({'first_name': 'Wonderful', 'last_name': 'Spam'})
Top answer
1 of 7
7

Using Miller (mlr), a CSV-aware multi-purpose processing utility for various structured document formats, to clean up the whitespace of all fields:

$ cat file
"host1","host1","linux
server",""
"host2","host2","linux server",""
$ mlr --csv -N clean-whitespace file
host1,host1,linux server,
host2,host2,linux server,

This reads the data in file as header-less CSV records and applies the clean-whitespace operation to each. The clean-whitespace operation trims flanking whitespace from each field's value and combines consecutive whitespace characters into single spaces.

To instead only replace newlines with spaces, you may iterate over the fields with a short put expression:

*) { $[k] = gssub(v, "\n", " ") }' file
host1,host1,linux server,
host2,host2,linux server,

The gssub() function acts like gsub() in Awk, but does not treat its query argument like a regular expression (Miller also has gsub()).

If you feel you need to have the fields quoted even though it's not strictly needed (Miller adds quotes automatically if a field's value requires it), then use mlr with its --quote-all option:

$ mlr --csv -N --quote-all clean-whitespace file
"host1","host1","linux server",""
"host2","host2","linux server",""
$ mlr --csv -N --quote-all put 'for (k,v in $*) { $[k] = gssub(v, "\n", " ") }' file
"host1","host1","linux server",""
"host2","host2","linux server",""
2 of 7
6

The last thing you want to do is try to do this in bash. See Why is using a shell loop to process text considered bad practice?.

Now, if what you want can be expressed as "remove any newline characters unless they come right after a " character", you could do something like this:

perl -pe 's/(?<!")\n/ /g' file

The (?<!")\n matches any newline character that is NOT preceded by a ". So given an example input like this:

$ cat file
"host0","host0","linux
server",""
"host1","host1","linux
server
centos",""
"host2","host2","linux server",""

The command above gives:

$ perl -pe 's/(?<!")\n/ /g' file
"host0","host0","linux server",""
"host1","host1","linux server centos",""
"host2","host2","linux server",""

But, really, mlr is the best approach.

🌐
Python
bugs.python.org › issue7198
Issue 7198: Extraneous newlines with csv.writer on Windows - Python tracker
This issue tracker has been migrated to GitHub, and is currently read-only. For more information, see the GitHub FAQs in the Python's Developer Guide · This issue has been migrated to GitHub: https://github.com/python/cpython/issues/51447
Find elsewhere
🌐
GitConnected
levelup.gitconnected.com › manage-newline-breaks-in-csv-files-with-python-160fefb9ac45
Manage newline breaks in CSV files with Python | by Stephen David-Williams | Level Up Coding
November 27, 2023 - Let’s explore simple solutions to help manage such scenarios effectively with Python. CSV files use specific characters (or delimiters), like commas (’,’) or pipes (’|’) to separate values and newlines to indicate new rows. But what if the data itself contains new lines (within a cell)? ... Notice the quotes used around the “Turned off at panel” cell. These exist because the CSV writer used quotes to handle cells with multiple cells i.e.
🌐
UiPath Community
forum.uipath.com › help › activities
Remove new lines in a csv - Help - Activities - UiPath Community Forum
October 13, 2022 - Hello guys I have a CSV input as attached. Problem is there are some lines where there is a new line added and should be removed. So that first column is always ID. Can this be done with UiPath studio activities? Any…
🌐
Reddit
reddit.com › r/learnpython › changing carriage return to newline in a csv file using python
r/learnpython on Reddit: Changing carriage return to newline in a csv file using python
November 12, 2019 -

For a uni assignment, I need to process a csv file. However these csv files were written such that the EOL character in every case is a carriage return (\r) character. The assignment prompt says that I cannot read in the file using the csv module as it will not recognise the \r character and thus the requirement of the assignment is that, in my Python script, I change the carriage return to a newline or any other EOL character that Python's csv module can recognise as a valid EOL. How would I do this? Also why doesn't the csv module recognise the carriage return? (Sorry for poor English and/or bad formatting)

🌐
Codecademy Forums
discuss.codecademy.com › computer science
What does the `newline=" "` argument do? - Computer Science - Codecademy Forums
January 30, 2019 - Could somebody please explain to me in layman’s terms, what the newline = " " argument does? Documentation is written in a language which only takes me down a rabbit hole of words I don’t understand yet.
🌐
GitHub
github.com › golang › go › issues › 36445
encoding/csv: writer.UseCRLF will change \n to \r\n in data field · Issue #36445 · golang/go
January 8, 2020 - into csv file · but the newline in asd\njk has been change to asd\r\njk · playground · \n in data field would not be changed by writer.UseCRLF · "col1,col2\r\n\"asd\njk\",2g9\r\n" "col1,col2\r\n\"asd\r\njk\",2g9\r\n" Reactions are currently unavailable ·
Author   golang
🌐
iO Flood
ioflood.com › blog › python-print-without-newline
[SOLVED] Python Print Without Newline? Syntax and Examples
February 14, 2024 - Python’s print function adds a newline character (‘\n’) by default at the end of the output. However, you can modify this behavior with the ‘end’ parameter. If you want to print without a newline, use an empty string with the ‘end’ parameter. For instance print('Hello, World!', end='').
🌐
Frankcorso
frankcorso.me › frank corso › reading from writing to csv files python
Reading From and Writing to <span class="caps">CSV</span> Files in Python - Frank Corso
May 12, 2025 - with open('somefile.csv', newline='', encoding='utf-8') as fh: # Do stuff · To get started, let’s do a simple exercise of writing a row to a new file and then reading that row. To work with CSV files, we first need to import the csv module. ... Then, we will use its writer() method to create a writer object that will perform the writing for us.
🌐
Zditect
zditect.com › blog › 55957398.html
Redirecting...
We cannot provide a description for this page right now
🌐
Python Tutorial
pythontutorial.net › home › python basics › python write csv file
How to Write to CSV Files in Python
March 30, 2025 - import csv header = ['name', 'area', ... line between two subsequent rows: To remove the blank line, you pass the keyword argument newline='' to the open() function as follows: import csv header = ['name', 'area', 'country_code2', ...
🌐
Codegrepper
codegrepper.com › code-examples › python › csv+writerow+without+newline
csv writerow without newline Code Example
November 11, 2021 - with open('output.csv', 'w', newline='\n', encoding='utf-8') as f: writer = csv.writer(f) ...
🌐
Adoclib
adoclib.com › blog › how-to-properly-remove-carriage-return-in-python-while-using-with-dictwriter-newline-does-not-help.html
ADocLib .NET SDKs for PDF, Excel, Word, OCR, Barcodes | Web, WinForms & WPF Development
Contact the Domain Owner: webmaster@adoclib.com provides high-performance .NET PDF SDK, .NET Word SDK, .NET Excel SDK, .NET OCR SDK, and .NET Barcodes SDK for Barcodes Generation & Recognition; free trials for evaluation; code samples for C# and VB.NET programmings.