March 23, 2024 - colmn=[ 'I','UNIT','XX','VERSION','SETTLEMENTDATE','RUNNO', 'DUID','INTERVENTION','DISPATCHMODE','AGCSTATUS','INITIALMW', 'TOTALCLEARED','RAMPDOWNRATE','RAMPUPRATE','LOWER5MIN', 'LOWER60SEC','LOWER6SEC','RAISE5MIN','RAISE60SEC', 'RAISE6SEC','MARGINAL5MINVALUE','MARGINAL60SECVALUE', 'MARGINAL6SECVALUE','MARGINALVALUE','VIOLATION5MINDEGREE' ] raw = pl.scan_csv(f'{raw_landing}/csv/*.CSV',skip_rows=1, new_columns=colmn,has_header=False, infer_schema_length=0,truncate_ragged_lines=True) transform =( raw .filter(pl.col("I")=='D') .filter(pl.col("UNIT")=='DUNIT') .filter(pl.col("VERSION")=='3') .drop("XX") .drop("I") ) No response · basically, there are more than 25 columns, but Polars so far seems to account only for the total columns founds at row =1 · Polars should read a csv with variable column numbers ·

October 11, 2023 - A-io-csvArea: reading/writing CSV filesArea: reading/writing CSV filesP-mediumPriority: mediumPriority: mediumbugSomething isn't workingSomething isn't workingpythonRelated to Python PolarsRelated to Python Polars ... I have checked that this issue has not already been reported. I have confirmed this bug exists on the latest version of Polars. import io import polars as pl csv = ''' # This is a comment A,B 1,Hello 2,World '''.strip() schema = { 'A': pl.Int32(), 'B': pl.Utf8(), } df = pl.read_csv(io.StringIO(csv), comment_char='#', schema=schema)

Read a CSV file or string into a Dataframe. ... Maximum number of lines to read to infer schema. If set to 0, all columns will be read as pl.Utf8.

pl$read_csv( source, ..., has_header = TRUE, separator = ",", comment_prefix = NULL, quote_char = "\"", skip_rows = 0, schema = NULL, schema_overrides = NULL, null_values = NULL, missing_utf8_is_empty_string = FALSE, ignore_errors = FALSE, cache = FALSE, infer_schema = TRUE, infer_schema_length ...

September 26, 2024 - Traceback (most recent call last): File "test_polars_bug.py", line 11, in <module> pl.read_csv(io.StringIO(csv), schema_overrides=s) File ".../lib/python3.10/site-packages/polars/_utils/deprecation.py", line 91, in wrapper return function(*args, **kwargs) File ".../lib/python3.10/site-packages/polars/_utils/deprecation.py", line 91, in wrapper return function(*args, **kwargs) File ".../lib/python3.10/site-packages/polars/_utils/deprecation.py", line 91, in wrapper return function(*args, **kwargs) File ".../lib/python3.10/site-packages/polars/io/csv/functions.py", line 500, in read_csv df = _read_csv_impl( File ".../lib/python3.10/site-packages/polars/io/csv/functions.py", line 646, in _read_csv_impl pydf = PyDataFrame.read_csv( polars.exceptions.ComputeError: found more fields than defined in 'Schema' Consider setting 'truncate_ragged_lines=True'.

July 31, 2025 - import io import polars as pl csv = """name,age John,20 Jane,30 """ df = pl.read_csv( io.StringIO(csv), columns=[1], new_columns=["age_with_tuple"], schema_overrides=(pl.String,) ) print(df) df = pl.read_csv( io.StringIO(csv), columns=[1], new_columns=["age_with_list"], schema_overrides=[pl.String] ) print(df) df = pl.read_csv( io.StringIO(csv), columns=[1], new_columns=["age_with_dict"], schema_overrides={"age_with_dict": pl.String} ) print(df) shape: (2, 1) ┌────────────────┐ │ age_with_tuple │ │ --- │ │ i64 │ ╞═══════════

February 2, 2024 - import polars as pl from datetime import date pl.DataFrame( { "a": [1, 2, 3], "b": ["a", "b", "c"], "c": [date(2024, 1, 1), date(2024, 1, 2), date(2024, 1, 3)], } ).write_csv("test.csv") df = pl.read_csv("test.csv", columns=["a", "c"], schema={"a": pl.Int32, "c": pl.Date}) polars.exceptions.ComputeError: could not parse `a` as dtype `date` at column 'c' (column number 2) When the columns parameter is specified, the schema parameter ignores the dictionary and attempts to apply the values in the schema dictionary to the original columns.

November 7, 2024 - This matches the behavior of schema_overrides in pl.read_csv, which does not require the entire schema to function. ... --------Version info--------- Polars: 1.12.0 Index type: UInt32 Platform: Windows-10-10.0.19045-SP0 Python: 3.12.7 (tags/v3.12.7:0b05ead, Oct 1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)] LTS CPU: False ----Optional dependencies---- adbc_driver_manager <not installed> altair <not installed> cloudpickle <not installed> connectorx <not installed> deltalake <not installed> fastexcel <not installed> fsspec <not installed> gevent <not installed> great_tables <not installed> matplotlib 3.9.2 nest_asyncio 1.6.0 numpy 1.26.4 openpyxl <not installed> pandas 2.2.3 pyarrow 17.0.0 pydantic <not installed> pyiceberg <not installed> sqlalchemy <not installed> torch <not installed> xlsx2csv <not installed> xlsxwriter <not installed>

December 15, 2023 - Reproducible example from io import StringIO import polars as pl csv = StringIO( "Name,Age,ID\n" "Mark,48,AAA1234\n" "Beth,26,BBB1234\n" "Steve,19,CCC1234\n" ) df = pl.read_csv( csv, columns=["Name", "Age", "ID"], new_columns=["Name2", "...

January 31, 2024 - CSV parsing fails with pl.read_csv(path, schema=schema) for a perfectly reasonable CSV file and schema. Note that the above MWE does work correctly if you use pl.scan_csv(bug_path, schema=schema).collect() instead. CSV parsing works. ... --------Version info--------- Polars: 0.20.6 Index type: UInt32 Platform: Windows-10-10.0.19044-SP0 Python: 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] ----Optional dependencies---- adbc_driver_manager: <not installed> cloudpickle: <not installed> connectorx: <not installed> deltalake: <not installed> fsspec: <not installed> gevent: <not installed> hvplot: <not installed> matplotlib: <not installed> numpy: 1.26.3 openpyxl: <not installed> pandas: 2.2.0 pyarrow: 15.0.0 pydantic: <not installed> pyiceberg: <not installed> pyxlsb: <not installed> sqlalchemy: <not installed> xlsx2csv: <not installed> xlsxwriter: 3.1.9

February 1, 2024 - I have confirmed this bug exists on the latest version of Polars. ... with open(filename, 'rb') as fin: with gzip.open(fin) as data: df = pl.read_csv( data, separator='|', has_header=False, new_columns=list(schema.keys()), schema=schema, )

Single byte character used for csv quoting, default = ". Set to None to turn off special handling and escaping of quotes. ... Start reading after skip_rows lines. ... Provide the schema. This means that polars doesn’t do schema inference.

January 27, 2025 - import polars as pl # 100 integers, then a float df = pl.DataFrame({"a": ["1"] * 100 + ["1.5"]}) df.write_csv("test_float.csv") # This is expected to fail--default schema inference is 100 rows # df = pl.read_csv("test_float.csv") # TypeError: unexpected value while building Series of type Int64; found value of type Float64: 1.5 # This should succeed, but does not--we tell it to read column a as a float.

January 24, 2025 - import io import polars as pl # a: int, b: string csv_body = """a,b\n1,hi""" # 1. Expected pl.read_csv(io.StringIO(csv_body), schema_overrides={'a': pl.Int64, 'b': pl.String}) # shape: (1, 2) # ┌─────┬─────┐ # │ a ┆ b │ # │ --- ┆ --- │ # │ i64 ┆ str │ # ╞═════╪═════╡ # │ 1 ┆ hi │ # └─────┴─────┘ # 2.

August 1, 2024 - It appears that the documentation for pl.read_csv() made a silent switch from infer_schema_length from 0 to False to read as strings. It seems that both are supported, but the docs no longer explain what infer_schema_length=0 means. ... https://docs.pola.rs/releases/upgrade/1/#properly-apply-strict-parameter-in-series-constructor and https://docs.pola.rs/api/python/version/0.20/reference/api/polars.read_csv.html#:~:text=of your system.-,infer_schema_length,-The maximum number

February 16, 2024 - import polars as pl import io csv_data = """ "Time","a","b","b","d", 0, 0.0,300.0, 50.5,Process, 0, 0.0,300.0, 50.4,Process, 1, 0.0,300.0, 50.5,Process, 2, 0.0,300.0, 50.5,Process, 3, 0.0,300.0, 50.5,Process, 5, 0.0,300.0, 50.5,Process, 5, 0.0,300.0, 50.5,Process, 6, 0.0,299.0, 50.5,Process, 7, 0.0,300.0, 50.5,Process, 9, 0.0,300.0, 50.5,Process, """ df =pl.read_csv(io.StringIO(csv_data)) print(df.dtypes)

September 18, 2024 - from pathlib import Path import polars as pl data = [ {'b': 1, 'a': 2, 'c': 3}, {'b': 1, 'a': 2, 'c': 3}, {'b': 1, 'a': 2, 'c': 3}, ] outpath = Path('/tmp') / 'test.csv' pl.DataFrame(data).write_csv(outpath) pl.read_csv(outpath) pl.read_csv(outpath, schema={'a': pl.Int16, 'b': pl.Int16, 'c': pl.Int16})

February 12, 2023 - Another thing that’s not directly related to reading csv is that Polars tells you columns types when you output your dataframe. Pandas doesn’t do this and I honestly love this feature! scan_csv() is more efficient way to read a csv file. The below is what the documentation says: This allows the query optimizer to push down predicates and projections to the scan level, thereby potentially reducing memory overhead. https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.scan_csv.html#polars.scan_csv

GitHub · Discord · Twitter · polars.read_csv_batched( source: str | Path, *, has_header: bool = True, columns: Sequence[int] | Sequence[str] | None = None, new_columns: Sequence[str] | None = None, separator: str = ',', comment_prefix: str | None = None, quote_char: str | None = '"', skip_rows: int = 0, schema_overrides: Mapping[str, PolarsDataType] | Sequence[PolarsDataType] | None = None, null_values: str | Sequence[str] | dict[str, str] | None = None, missing_utf8_is_empty_string: bool = False, ignore_errors: bool = False, try_parse_dates: bool = False, n_threads: int | None = None, infe

Polars has its own fast implementation for csv reading with many flexible configuration options. ... let mut file = File::create("output.csv").expect("could not create file"); CsvWriter::new(&mut file).has_header(true).with_delimiter(b',').finish(&mut df); let df_csv = CsvReader::from_path...