Here is a simple solution: Replace all \n with \\n before saving to CSV. This will preserve the newline characters.

df.loc[:, "Column_Name"] = df["Column_Name"].apply(lambda x: x.replace('\n', '\\n'))
df.to_csv("df.csv", index=False)
Answer from betelgeuse on Stack Exchange
Top answer
1 of 2
19

Here is a simple solution: Replace all \n with \\n before saving to CSV. This will preserve the newline characters.

df.loc[:, "Column_Name"] = df["Column_Name"].apply(lambda x: x.replace('\n', '\\n'))
df.to_csv("df.csv", index=False)
2 of 2
10

I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, although the principle could be applied to Python 2.

The main trick

This is to replace the \n characters before writing with a weird character that otherwise wouldn't be included, then to swap that weird character back for \n after reading the file back from disk.

For my weird character, I will use the Icelandic thorn: Þ, but you can choose anything that should otherwise not appear in your text variables. Its name, as defined in the standardised Unicode specification is: LATIN SMALL LETTER THORN. You can use it in Python 3 a couple of ways:

    weird_literal = 'þ'
    weird_name = '\N{LATIN SMALL LETTER THORN}'
    weird_char = '\xfe'  # hex representation
    weird_literal == weird_name == weird_char  # True

That \N is pretty cool (and works in python 3.6 inside formatted strings too)... it basically allows you to pass the Name of a character, as per Unicode's specification.

An alternative character that may serve as a good standard is '\u2063' (INVISIBLE SEPARATOR).

Replacing \n

Now we use this weird character to replace '\n'. Here are the two ways that pop into my mind for achieving this:

  1. using a list comprehension on your list of lists: data:

     new_data = [[sample[0].replace('\n', weird_char) + weird_char, sample[1]]
                  for sample in data]
    
  2. putting the data into a dataframe, and using replace on the whole text column in one go

     df1 = pd.DataFrame(data, columns=['text', 'category'])
     df1.text = df.text.str.replace('\n', weird_char)
    

The resulting dataframe looks like this, with newlines replaced:

               text              category
0         some text in one line      1   
1  text withþnew line character      0   
2    another newþline character      1   

Writing the results to disk

Now we write either of those identical dataframes to disk. I set index=False as you said you don't want row numbers to be in the CSV:

FILE = '~/path/to/test_file.csv'
df.to_csv(FILE, index=False)

What does it look like on disk?

text,category

some text in one line,1

text withþnew line character,0

another newþline character,1

Getting the original data back from disk

Read the data back from file:

new_df = pd.read_csv(FILE)

And we can replace the Þ characters back to \n:

new_df.text = new_df.text.str.replace(weird_char, '\n')

And the final DataFrame:

new_df
               text               category
0          some text in one line      1   
1  text with\nnew line character      0   
2    another new\nline character      1   

If you want things back into your list of lists, then you can do this:

original_lists = [[text, category] for index, text, category in old_df_again.itertuples()]

Which looks like this:

[['some text in one line', 1],
 ['text with\nnew line character', 0],
 ['another new\nline character', 1]]
Discussions

csv writer: append doesn't go to new line
The problem is not in your code, it's in your CSV file. text files (including CSV files) traditionally end in a newline, so that it's ready for data to be appended to it. For some reason your csv is breaking that tradition, and that's why you see this error. Easiest solution is to edit your CSV file and add the newline to the end. More on reddit.com
🌐 r/learnpython
3
2
January 26, 2021
Add new line to output in csv file in python - Stack Overflow
I am a newbie at Python & I have a web scraper program that retrieved links and puts them into a .csv file. I need to add a new line after each web link in the output but I do not know how to u... More on stackoverflow.com
🌐 stackoverflow.com
December 30, 2017
Python, write a new line in a CSV-File - Stack Overflow
I'm working on a Python-Script which takes some weather information from a Website and put it in a CSV-File every day. I want to append the new information below and not directly behind it. This ... More on stackoverflow.com
🌐 stackoverflow.com
November 1, 2016
Python inserts newline by writing to csv - Data Science Stack Exchange
I am trying to scrape http://www.the-numbers.com/movie/budgets/all but when I write the table into a csv file, there is an additional line with the counter index written in between each movie row.... More on datascience.stackexchange.com
🌐 datascience.stackexchange.com
🌐
Quora
quora.com › How-do-I-insert-a-new-line-when-adding-to-a-CSV-file-in-Python
How to insert a new line when adding to a CSV file in Python - Quora
Answer: [code]import csv # Open the CSV file in append mode with open('file.csv', 'a', newline='') as csvfile: # Create a CSV writer object writer = csv.writer(csvfile) # Write the new line to the CSV file writer.writerow(['value1', 'value2', 'value3']) [/code]I hope this was he...
🌐
Reddit
reddit.com › r/learnpython › csv writer: append doesn't go to new line
r/learnpython on Reddit: csv writer: append doesn't go to new line
January 26, 2021 -

So, basically, I've got this code:

new_list_csv = []

def add_it():
    #gets the values from Entry
    #three different tk Entrys generate three different values
    name= self.e.get()
    ex= self.e_x.get()
    ey= self.e_y.get()
    #adds them to the new list
    new_list_csv.append(name)
    new_list_csv.append(ex)
    new_list_csv.append(ey)
    #appends new characters to csv file
    with open ('chr_list_copy.txt', 'a', newline='') as write_obj:   
        csv_writer = csv.writer(write_obj)
        csv_writer.writerow(new_list_csv)

and the csv file looks like this:

name,x,y
Ergo,1,1
Sum,5,5
Name12,1,2
Name34,3,4
Name56,5,6
Name78,7,8
Name910,9,10
Name1112,11,12
Name1314,13,14

if I try and add [Otto,6,9], I get:

name,x,y
[...]
Name1314,13,14Otto,6,9

instead of:

name,x,y
[...]
Name1314,13,14
Otto,6,9

I've used this very same structure for the rest of my code, but in this specific instance, it doesn't work.

🌐
GitConnected
levelup.gitconnected.com › manage-newline-breaks-in-csv-files-with-python-160fefb9ac45
Manage newline breaks in CSV files with Python | by Stephen David-Williams | Level Up Coding
November 27, 2023 - df["Description"] = df["Description"].str.replace("\\r\\n", "<br>") df.to_csv("new_file.csv", sep='|', index=False) ... The .str.replace("\\r\\n", "<br>") line replaces the newlines with an HTML break indicator, which represents the custom encoding ...
🌐
YouTube
youtube.com › codemade
how to add new line in csv file python - YouTube
Instantly Download or Run the code at https://codegive.com certainly! adding a new line to a csv file in python can be achieved using the csv module. here's...
Published   February 23, 2024
Views   128
Find elsewhere
🌐
Python
docs.python.org › 3 › library › csv.html
csv — CSV File Reading and Writing
Each row read from the csv file is returned as a list of strings. No automatic data type conversion is performed unless the QUOTE_NONNUMERIC format option is specified (in which case unquoted fields are transformed into floats). ... >>> import csv >>> with open('eggs.csv', newline='') as csvfile: ...
🌐
Stack Overflow
stackoverflow.com › questions › 40362824 › python-write-a-new-line-in-a-csv-file
Python, write a new line in a CSV-File - Stack Overflow
November 1, 2016 - def save_data(weatherdatasaved): filename = "weather.csv" header="Datum;Luft-Min;Luft-Max;Durchschn.Wassertemp;Barometer"+"\n" #Check whether the file exists if(os.path.exists(os.path.expanduser(filename))==False): file = open(os.path.expanduser(filename),"wb") file.write(bytes(header, encoding="ascii",errors="ignore")) file.close() file = open(os.path.expanduser(filename),"a") i=0 while (i < len(weatherdatasaved)): if((i+1)==len(weatherdatasaved): file.write(weatherdatasaved[i]+"\n") else: file.write(weatherdatasaved[i]+";") i+=1 file.close()
🌐
Stack Exchange
datascience.stackexchange.com › questions › 24670 › python-inserts-newline-by-writing-to-csv
Python inserts newline by writing to csv - Data Science Stack Exchange
I dont understand how that counter line is being written to the csv... import csv,os from bs4 import BeautifulSoup from urllib.request import Request, urlopen, URLError from selenium import webdriver counter = 0 currentDir=os.getcwd() filename = currentDir + "\\theNumbersScraper.csv" pagecount = 1 headers=['ID', 'Release Date', 'Movie', 'Production Budget', 'Domestic Gross', 'Worldwide Gross'] with open(filename, 'w' ,newline='\n',encoding='utf-8') as csvfile: #writer = csv.DictWriter(csvfile, fieldnames=dictionary)#write headers #writer.writeheader() #csvfile = open(filename, 'w', newline='',
🌐
Medium
medium.com › @richard.jones › unusual-newlines-in-csvs-888d2da838f4
Unusual newlines in CSVs. We were debugging an issue with a… | by Richard D Jones | Medium
June 11, 2018 - It’ll happily write your Paragraph Separator newline character into the file in an unquoted string, and your equivalent reader will not be able to read the row back. ... As the csv writer does not recognise \u2029 as a newline character, it does not wrap the quotes around the string like it would if it contained\n. The fix is easy, when constructing your writer in python you should do it like this:
🌐
Reddit
reddit.com › r/learnpython › changing carriage return to newline in a csv file using python
r/learnpython on Reddit: Changing carriage return to newline in a csv file using python
November 12, 2019 -

For a uni assignment, I need to process a csv file. However these csv files were written such that the EOL character in every case is a carriage return (\r) character. The assignment prompt says that I cannot read in the file using the csv module as it will not recognise the \r character and thus the requirement of the assignment is that, in my Python script, I change the carriage return to a newline or any other EOL character that Python's csv module can recognise as a valid EOL. How would I do this? Also why doesn't the csv module recognise the carriage return? (Sorry for poor English and/or bad formatting)

Top answer
1 of 6
540

Python 3:

The official csv documentation recommends opening the file with newline='' on all platforms to disable universal newlines translation:

with open('output.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    ...

The CSV writer terminates each line with the lineterminator of the dialect, which is '\r\n' for the default excel dialect on all platforms because that's what RFC 4180 recommends.


Python 2:

On Windows, always open your files in binary mode ("rb" or "wb"), before passing them to csv.reader or csv.writer.

Although the file is a text file, CSV is regarded a binary format by the libraries involved, with \r\n separating records. If that separator is written in text mode, the Python runtime replaces the \n with \r\n, hence the \r\r\n observed in the file.

See this previous answer.

2 of 6
291

While @john-machin gives a good answer, it's not always the best approach. For example, it doesn't work on Python 3 unless you encode all of your inputs to the CSV writer. Also, it doesn't address the issue if the script wants to use sys.stdout as the stream.

I suggest instead setting the 'lineterminator' attribute when creating the writer:

import csv
import sys

doc = csv.writer(sys.stdout, lineterminator='\n')
doc.writerow('abc')
doc.writerow(range(3))

That example will work on Python 2 and Python 3 and won't produce the unwanted newline characters. Note, however, that it may produce undesirable newlines (omitting the LF character on Unix operating systems).

In most cases, however, I believe that behavior is preferable and more natural than treating all CSV as a binary format. I provide this answer as an alternative for your consideration.

🌐
Python Forum
python-forum.io › thread-29412.html
Add a new line to a CSV in one column
Hi, I have a report that is stored in a list. One of the columns will return multiple entries. I would like to a new line between each entry. Is this possible? My report code is with open(full_report_path, 'w') as f: writer = csv.write...
🌐
Python Morsels
pythonmorsels.com › csv-writing
Writing a CSV file - Python Morsels
February 14, 2023 - Let's talk about creating CSV files in Python. Python's csv module has a writer callable, which accepts a file object or a file-like object, and which returns a writer object (similar to csv.reader): >>> import csv >>> csv_file = open("pets.csv", mode="wt", newline="") >>> writer = csv.writer(csv_file)
🌐
TheCodeForge
thecodeforge.io › home › python › csv in python — newline modes and silent data corruption
Working with CSV in Python: CSV in Python — Newline Modes | TheCodeForge
March 5, 2026 - This is why you must always open CSV files with newline=''—otherwise, csv.reader or csv.writer will mangle multi-line fields, and you'll lose data without any error. The same trap applies to encoding: Python 3's default system encoding may ...