🌐
Rho Signal
rhosignal.com › posts › reading-from-s3-with-filters
Reading and writing files on S3 with Polars | Rho Signal
December 22, 2024 - In this post we see how to read and write from a CSV or Parquet file from cloud storage with Polars. We also see how Polars applies query optimisations to reduce the amount of data transferred across the network. We use S3 as the example in this case but the same principles apply to other cloud ...
🌐
Polars
docs.pola.rs › user-guide › io › cloud-storage
Cloud storage - Polars user guide
Polars can read and write to AWS S3, Azure Blob Storage and Google Cloud Storage. The API is the same for all three storage providers. To read from cloud storage, additional dependencies may be needed depending on the use case and cloud storage provider: ... ParquetReader · CsvReader ·
Discussions

`read_csv` does not work on S3 path while `scan_csv` does
I expect to be able to read from S3 using read_csv as I do with scan_csv. ... --------Version info--------- Polars: 1.26.0 Index type: UInt32 Platform: Linux-5.10.234-225.910.amzn2.x86_64-x86_64-with-glibc2.26 Python: 3.9.10 (main, Jun 2 2022, 18:40:40) [GCC 7.3.1 20180712 (Red Hat 7.3.1-14)] ... More on github.com
🌐 github.com
4
March 28, 2025
`read_csv` from s3 fails when using `new_columns` on a csv without header
Checks I have checked that this issue has not already been reported. I have confirmed this bug exists on the latest version of Polars. Reproducible example import polars as pl # A file looking like this can be read from local but not fro... More on github.com
🌐 github.com
0
April 18, 2025
python - Polars reading just one file from s3 with glob patterns - Stack Overflow
I have a s3 location in which I have a list of directories and each directory contains a csv named sample_file.csv. I am trying to read these files using a glob pattern in pl.read_csv but it is just reading only one file and silently ignoring the rest. The issue has been mentioned in the polars git ... More on stackoverflow.com
🌐 stackoverflow.com
python - How to read parquet files from AWS S3 with polars - Stack Overflow
I am also successfully able to read the parquet using pandas just setting the AWS_PROFILE env var. Am I using the storage_options incorrectly? It doesn't seem able to take 'aws_profile' key value pair to extract local config credentials its self? ... Save this answer. ... Show activity on this post. You might also need to pass the corresponding session token as follows. Copyimport polars as pl import boto3 profile_name = "your-profile" s3... More on stackoverflow.com
🌐 stackoverflow.com
🌐
Reddit
reddit.com › r/dataengineering › read s3 data using polars
r/dataengineering on Reddit: Read S3 data using Polars
January 31, 2026 - If you have the opportunity to convert these files to parquet or ipc files, Polars will stream them directly from s3. ... Duckdb saved my life on my local computer. It has a very very good csv reader.
🌐
Medium
medium.com › better-programming › read-from-and-write-to-amazon-s3-in-polars-b0402e74b339
How to Read from and Write to Amazon S3 in Polars | by Yuki Kakegawa | Better Programming
June 24, 2023 - What you’d do is read a file in S3 through s3fs as pyarrow dataset. Then you convert it to a Polars dataframe (Make sure you have the necessary configurations for s3fs to work, such as setting up and specifying the IAM profile for AWS).
🌐
GitHub
github.com › pola-rs › polars › issues › 21970
`read_csv` does not work on S3 path while `scan_csv` does · Issue #21970 · pola-rs/polars
March 28, 2025 - session = boto3.session.Session() credentials = session.get_credentials() df = pl.read_csv( source="s3://bucket/file.csv", storage_options={ "aws_access_key_id": credentials.access_key, "aws_secret_access_key": credentials.secret_key, "session_token": credentials.token, "region": "eu-central-1", }, ) File "/.pyenv/versions/python39_loaded/lib/python3.9/site-packages/polars/io/csv/functions.py", line 532, in read_csv -- with prepare_file_arg( File "/.pyenv/versions/python39_loaded/lib/python3.9/site-packages/fsspec/core.py", line 103, in __enter__ f = self.fs.open(self.path, mode=mode) File "/.
Author   pola-rs
🌐
Medium
medium.com › @louis_10840 › how-to-process-data-stored-in-amazon-s3-using-polars-2305bf064c52
How to read data stored in Amazon S3 with Polars | by Louis Giron | Medium
March 1, 2024 - How to read data stored in Amazon S3 with Polars This article focuses on how to use Polars library with data stored in Amazon S3 for large-scale data processing. Amazon S3 Amazon Simple Storage …
🌐
Stuff by Yuki
stuffbyyuki.com › home › read from and write to amazon s3 in polars
Read from and Write to Amazon S3 in Polars - Stuff by Yuki
June 18, 2023 - What you’d basically do is to read a file in S3 through s3fs as pyarrow dataset, and then you convert it to a Polars dataframe (Make sure you have necessary configurations for s3fs to work, such as setting up and specifying IAM profile for AWS).
🌐
GitHub
github.com › pola-rs › polars › issues › 22334
`read_csv` from s3 fails when using `new_columns` on a csv without header · Issue #22334 · pola-rs/polars
April 18, 2025 - import polars as pl # A file looking like this can be read from local but not from S3 if specifying new_columns """ 1,0 2,0 3,0 """ # This works csv_path = "file.csv" df = pl.scan_csv( csv_file, has_header=False, # new_columns=["one", "another"], ).collect() print(df) # This works csv_path = "file.csv" df = pl.scan_csv( csv_file, has_header=False, new_columns=["one", "another"], ).collect() print(df) # This works csv_path = "s3://bucket/file.csv" df = pl.scan_csv( csv_file, has_header=False, # new_columns=["one", "another"], ).collect() print(df) # This does not work csv_path = "s3://bucket/file.csv" df = pl.scan_csv( csv_file, has_header=False, new_columns=["one", "another"], ).collect() print(df)
Author   pola-rs
Find elsewhere
🌐
Pola-rs
pola-rs.github.io › polars-book › user-guide › io › aws
AWS - Polars documentation
use aws_sdk_s3::Region; use aws_config::meta::region::RegionProviderChain; use aws_sdk_s3::Client; use std::borrow::Cow; use polars::prelude::*; #[tokio::main] async fn main() { let bucket = "<YOUR_BUCKET>"; let path = "<YOUR_PATH>"; let config = aws_config::from_env().load().await; let client = Client::new(&config); let req = client.get_object().bucket(bucket).key(path); let res = req.clone().send().await.unwrap(); let bytes = res.body.collect().await.unwrap(); let bytes = bytes.into_bytes(); let cursor = std::io::Cursor::new(bytes); let df = CsvReader::new(cursor).finish().unwrap(); println!("{:?}", df); }
🌐
Rho Signal
rhosignal.com › posts › reading-from-s3-with-sso
Reading from S3 with Polars (or DeltaLake) using AWS SSO | Rho Signal
December 22, 2024 - Polars can read and write files from S3. However, to do this Polars needs to authenticate into your AWS account. While there is a crude solution where we copy our AWS access key and secret key from the .aws/credentials file directly into Polars, this does not work with AWS Single Sign On (SSO).
🌐
GitHub
github.com › pola-rs › polars › issues › 13115
`scan_csv` does not work on s3 path while `read_csv` works · Issue #13115 · pola-rs/polars
December 19, 2023 - A-io-csvArea: reading/writing CSV filesArea: reading/writing CSV filesbugSomething isn't workingSomething isn't workingneeds triageAwaiting prioritization by a maintainerAwaiting prioritization by a maintainerpythonRelated to Python PolarsRelated to Python Polars ... I have checked that this issue has not already been reported. I have confirmed this bug exists on the latest version of Polars. some_s3_file = f"s3://{BUCKET}/data.csv" pl.read_csv(some_s3_file) # works pl.scan_csv(some_s3_file) # doesn't work
Author   pola-rs
🌐
Polars
docs.pola.rs › user-guide › getting-started
Getting started - Polars user guide
... cargo add polars -F lazy # Or Cargo.toml [dependencies] polars = { version = "x", features = ["lazy", ...]} Polars supports reading and writing for common file formats (e.g., csv, json, parquet), cloud storage (S3, Azure Blob, BigQuery) and databases (e.g., postgres, mysql).
🌐
LinkedIn
linkedin.com › posts › daniel-beach-6ab8b4132_pandas-pyarrow-polars-activity-7088162732587323392-Emug
Daniel Beach on LinkedIn: #pandas #pyarrow #polars #aws #lambda #csv #s3 #3sfs #pyarrow #polars #s3…
July 21, 2023 - Hi Daniel Beach Polars has scan_pyarrow_dataset, and Pyarrow datasets also supports CSV https://arrow.apache.org/docs/python/dataset.html So yes it should be possible I use scan_pyarrow_dataset regularly for Parquet files ... Sr. Advisory Data Engineer @ SCE | Cloud Agnostic Data Engineering | Palantir Foundry 6mo ... #Parquet format (and it’s subsequent drivers in different libraries) is designed to read multi part file from a folder.
🌐
Kedro
docs.kedro.org › projects › kedro-datasets › en › kedro-datasets-3.0.0.post1 › _modules › kedro_datasets › polars › csv_dataset.html
kedro_datasets.polars.csv_dataset — kedro-datasets 0.19.5 documentation
code-block:: yaml cars: type: polars.CSVDataset filepath: data/01_raw/company/cars.csv load_args: sep: "," parse_dates: False save_args: has_header: False null_value: "somenullstring" motorbikes: type: polars.CSVDataset filepath: s3://your_bucket/data/02_intermediate/company/motorbikes.csv credentials: dev_s3 Example usage for the `Python API <https://kedro.readthedocs.io/en/stable/data/\ advanced_data_catalog_usage.html>`_: .. code-block:: pycon >>> from kedro_datasets.polars import CSVDataset >>> import polars as pl >>> >>> data = pl.DataFrame({"col1": [1, 2], "col2": [4, 5], "col3": [5, 6]}) >>> >>> dataset = CSVDataset(filepath=tmp_path / "test.csv") >>> dataset.save(data) >>> reloaded = dataset.load() >>> assert data.frame_equal(reloaded) """ DEFAULT_LOAD_ARGS: dict[str, Any] = {"rechunk": True} DEFAULT_SAVE_ARGS: dict[str, Any] = {}
🌐
Rho Signal
rhosignal.com › posts › polars-glob-csvs
To go big you must be lazy | Rho Signal
December 22, 2024 - Polars can read and write files from S3. However, to do this Polars needs to authenticate into your AWS account.