Rho Signal
rhosignal.com › posts › reading-from-s3-with-filters
Reading and writing files on S3 with Polars | Rho Signal
December 22, 2024 - In this post we see how to read and write from a CSV or Parquet file from cloud storage with Polars. We also see how Polars applies query optimisations to reduce the amount of data transferred across the network. We use S3 as the example in this case but the same principles apply to other cloud ...
Polars
docs.pola.rs › user-guide › io › cloud-storage
Cloud storage - Polars user guide
Polars can read and write to AWS S3, Azure Blob Storage and Google Cloud Storage. The API is the same for all three storage providers. To read from cloud storage, additional dependencies may be needed depending on the use case and cloud storage provider: ... ParquetReader · CsvReader ·
`read_csv` does not work on S3 path while `scan_csv` does
I expect to be able to read from S3 using read_csv as I do with scan_csv. ... --------Version info--------- Polars: 1.26.0 Index type: UInt32 Platform: Linux-5.10.234-225.910.amzn2.x86_64-x86_64-with-glibc2.26 Python: 3.9.10 (main, Jun 2 2022, 18:40:40) [GCC 7.3.1 20180712 (Red Hat 7.3.1-14)] ... More on github.com
`read_csv` from s3 fails when using `new_columns` on a csv without header
Checks I have checked that this issue has not already been reported. I have confirmed this bug exists on the latest version of Polars. Reproducible example import polars as pl # A file looking like this can be read from local but not fro... More on github.com
python - Polars reading just one file from s3 with glob patterns - Stack Overflow
I have a s3 location in which I have a list of directories and each directory contains a csv named sample_file.csv. I am trying to read these files using a glob pattern in pl.read_csv but it is just reading only one file and silently ignoring the rest. The issue has been mentioned in the polars git ... More on stackoverflow.com
python - How to read parquet files from AWS S3 with polars - Stack Overflow
I am also successfully able to read the parquet using pandas just setting the AWS_PROFILE env var. Am I using the storage_options incorrectly? It doesn't seem able to take 'aws_profile' key value pair to extract local config credentials its self? ... Save this answer. ... Show activity on this post. You might also need to pass the corresponding session token as follows. Copyimport polars as pl import boto3 profile_name = "your-profile" s3... More on stackoverflow.com
Reddit
reddit.com › r/dataengineering › read s3 data using polars
r/dataengineering on Reddit: Read S3 data using Polars
January 31, 2026 - If you have the opportunity to convert these files to parquet or ipc files, Polars will stream them directly from s3. ... Duckdb saved my life on my local computer. It has a very very good csv reader.
BIG Data help - Polars and Arrow implementation on many (100TB) of parquet files. Mar 1, 2024
r/rstats 2y ago
What resources or tutorials helped you get the most advanced knowledge of Polars? Jan 21, 2026
r/dataengineering 6mo ago
GitHub
github.com › pola-rs › polars › issues › 21970
`read_csv` does not work on S3 path while `scan_csv` does · Issue #21970 · pola-rs/polars
March 28, 2025 - session = boto3.session.Session() credentials = session.get_credentials() df = pl.read_csv( source="s3://bucket/file.csv", storage_options={ "aws_access_key_id": credentials.access_key, "aws_secret_access_key": credentials.secret_key, "session_token": credentials.token, "region": "eu-central-1", }, ) File "/.pyenv/versions/python39_loaded/lib/python3.9/site-packages/polars/io/csv/functions.py", line 532, in read_csv -- with prepare_file_arg( File "/.pyenv/versions/python39_loaded/lib/python3.9/site-packages/fsspec/core.py", line 103, in __enter__ f = self.fs.open(self.path, mode=mode) File "/.
Author pola-rs
GitHub
github.com › pola-rs › polars › issues › 22334
`read_csv` from s3 fails when using `new_columns` on a csv without header · Issue #22334 · pola-rs/polars
April 18, 2025 - import polars as pl # A file looking like this can be read from local but not from S3 if specifying new_columns """ 1,0 2,0 3,0 """ # This works csv_path = "file.csv" df = pl.scan_csv( csv_file, has_header=False, # new_columns=["one", "another"], ).collect() print(df) # This works csv_path = "file.csv" df = pl.scan_csv( csv_file, has_header=False, new_columns=["one", "another"], ).collect() print(df) # This works csv_path = "s3://bucket/file.csv" df = pl.scan_csv( csv_file, has_header=False, # new_columns=["one", "another"], ).collect() print(df) # This does not work csv_path = "s3://bucket/file.csv" df = pl.scan_csv( csv_file, has_header=False, new_columns=["one", "another"], ).collect() print(df)
Author pola-rs
Pola-rs
pola-rs.github.io › polars-book › user-guide › io › aws
AWS - Polars documentation
use aws_sdk_s3::Region; use aws_config::meta::region::RegionProviderChain; use aws_sdk_s3::Client; use std::borrow::Cow; use polars::prelude::*; #[tokio::main] async fn main() { let bucket = "<YOUR_BUCKET>"; let path = "<YOUR_PATH>"; let config = aws_config::from_env().load().await; let client = Client::new(&config); let req = client.get_object().bucket(bucket).key(path); let res = req.clone().send().await.unwrap(); let bytes = res.body.collect().await.unwrap(); let bytes = bytes.into_bytes(); let cursor = std::io::Cursor::new(bytes); let df = CsvReader::new(cursor).finish().unwrap(); println!("{:?}", df); }
Rho Signal
rhosignal.com › posts › reading-from-s3-with-sso
Reading from S3 with Polars (or DeltaLake) using AWS SSO | Rho Signal
December 22, 2024 - Polars can read and write files from S3. However, to do this Polars needs to authenticate into your AWS account. While there is a crude solution where we copy our AWS access key and secret key from the .aws/credentials file directly into Polars, this does not work with AWS Single Sign On (SSO).
GitHub
github.com › pola-rs › polars › issues › 13115
`scan_csv` does not work on s3 path while `read_csv` works · Issue #13115 · pola-rs/polars
December 19, 2023 - A-io-csvArea: reading/writing CSV filesArea: reading/writing CSV filesbugSomething isn't workingSomething isn't workingneeds triageAwaiting prioritization by a maintainerAwaiting prioritization by a maintainerpythonRelated to Python PolarsRelated to Python Polars ... I have checked that this issue has not already been reported. I have confirmed this bug exists on the latest version of Polars. some_s3_file = f"s3://{BUCKET}/data.csv" pl.read_csv(some_s3_file) # works pl.scan_csv(some_s3_file) # doesn't work
Author pola-rs
Polars
docs.pola.rs › user-guide › getting-started
Getting started - Polars user guide
... cargo add polars -F lazy # Or Cargo.toml [dependencies] polars = { version = "x", features = ["lazy", ...]} Polars supports reading and writing for common file formats (e.g., csv, json, parquet), cloud storage (S3, Azure Blob, BigQuery) and databases (e.g., postgres, mysql).
Polars
docs.pola.rs › docs › python › dev › reference › api › polars.read_csv.html
polars.read_csv — Polars documentation
Lazily read from a CSV file or multiple files via glob patterns.
LinkedIn
linkedin.com › posts › daniel-beach-6ab8b4132_pandas-pyarrow-polars-activity-7088162732587323392-Emug
Daniel Beach on LinkedIn: #pandas #pyarrow #polars #aws #lambda #csv #s3 #3sfs #pyarrow #polars #s3…
July 21, 2023 - Hi Daniel Beach Polars has scan_pyarrow_dataset, and Pyarrow datasets also supports CSV https://arrow.apache.org/docs/python/dataset.html So yes it should be possible I use scan_pyarrow_dataset regularly for Parquet files ... Sr. Advisory Data Engineer @ SCE | Cloud Agnostic Data Engineering | Palantir Foundry 6mo ... #Parquet format (and it’s subsequent drivers in different libraries) is designed to read multi part file from a folder.
Kedro
docs.kedro.org › projects › kedro-datasets › en › kedro-datasets-3.0.0.post1 › _modules › kedro_datasets › polars › csv_dataset.html
kedro_datasets.polars.csv_dataset — kedro-datasets 0.19.5 documentation
code-block:: yaml cars: type: polars.CSVDataset filepath: data/01_raw/company/cars.csv load_args: sep: "," parse_dates: False save_args: has_header: False null_value: "somenullstring" motorbikes: type: polars.CSVDataset filepath: s3://your_bucket/data/02_intermediate/company/motorbikes.csv credentials: dev_s3 Example usage for the `Python API <https://kedro.readthedocs.io/en/stable/data/\ advanced_data_catalog_usage.html>`_: .. code-block:: pycon >>> from kedro_datasets.polars import CSVDataset >>> import polars as pl >>> >>> data = pl.DataFrame({"col1": [1, 2], "col2": [4, 5], "col3": [5, 6]}) >>> >>> dataset = CSVDataset(filepath=tmp_path / "test.csv") >>> dataset.save(data) >>> reloaded = dataset.load() >>> assert data.frame_equal(reloaded) """ DEFAULT_LOAD_ARGS: dict[str, Any] = {"rechunk": True} DEFAULT_SAVE_ARGS: dict[str, Any] = {}
LinkedIn
linkedin.com › posts › liam-brannigan-9080b214a_reading-and-writing-files-on-s3-with-polars-activity-7265379211798876162-lTEd
Reading and writing files on S3 with Polars | Liam Brannigan
We cannot provide a description for this page right now
Rho Signal
rhosignal.com › posts › polars-glob-csvs
To go big you must be lazy | Rho Signal
December 22, 2024 - Polars can read and write files from S3. However, to do this Polars needs to authenticate into your AWS account.