As jqurious suggested the default behavior of Polars write_csv is to cast a Polars df to a string, if not output file is given:
polarsdf.write_csv(separator='\t')
And for Shiny for Python this is my solution now:
def download_something():
yield polarsdf.write_csv(separator='\t')
Answer from gernophil on Stack OverflowTo append to a CSV file for example - you can pass a file object e.g.
import polars as pl
df1 = pl.DataFrame({"a": [1, 2], "b": [3 ,4]})
df2 = pl.DataFrame({"a": [5, 6], "b": [7 ,8]})
with open("out.csv", mode="a") as f:
df1.write_csv(f)
df2.write_csv(f, include_header=False)
>>> from pathlib import Path
>>> print(Path("out.csv").read_text(), end="")
a,b
1,3
2,4
5,7
6,8
Further discussion: "Append on export"
- https://github.com/pola-rs/polars/issues/21880
You can also convert Polars df to pandas df using .to_pandas() method, and then save to csv with mode="a+".
Polars already parses csv files as fast as it can. It already uses all threads, so trying to spin separate processes will only hurt performance.
You can set rechunk=False. That will save an expensive reallocation after reading.
Other things you can try is set the schema to smaller data types, so that you will have less memory and cache pressure.
If you have a say on how this file is written, you should go/ask for a parquet file. That will reduce file size and also speed up read, especially if you do not need all columns of the dataset.