It's hard to answer this question as I don't know what your other df looks like and I don't know exactly what you're doing that crashes your computer but here are a few observations/suggestions.

arrow and, by extension, polars isn't optimized for strings so one of the worst things you could do is load a giant file with all the columns being loaded as strings.

From the scan_csv docs

infer_schema_length Maximum number of lines to read to infer schema. If set to 0, all columns will be read as pl.Utf8.

You are setting that to 0 which means you're loading a giant file as all strings. Don't don't do that. Either set it to something reasonable or manually set the dtypes parameter. Ideally, the latter. Additionally, if there are columns that are strings but have relatively few unique values then you can make them categoricals to save more memory. Also, settings your integers to the smallest variation will also save some memory. One issue with Categoricals is that it's trickier to join on those columns. To get around that try:

import polars as pl
with pl.StringCache():
    dtypes={
    'siren':pl.UInt32(),
    'nic':pl.UInt32(),
    'siret':pl.UInt64(),
    'dateFin':pl.Date(),
    'dateDebut':pl.Date(),
    'etatAdministratifEtablissement':pl.Categorical(),
    'changementEtatAdministratifEtablissement':pl.Boolean(),
    'enseigne1Etablissement':pl.Categorical(),
    'enseigne2Etablissement':pl.Categorical(),
    'enseigne3Etablissement':pl.Categorical(),
    'changementEnseigneEtablissement':pl.Boolean(),
    'denominationUsuelleEtablissement':pl.Categorical(),
    'changementDenominationUsuelleEtablissement':pl.Boolean(),
    'activitePrincipaleEtablissement':pl.Categorical(),
    'nomenclatureActivitePrincipaleEtablissement':pl.Categorical(),
    'changementActivitePrincipaleEtablissement':pl.Boolean(),
    'caractereEmployeurEtablissement':pl.Categorical(),
    'changementCaractereEmployeurEtablissement':pl.Boolean()
    }

    df=pl.scan_csv("StockEtablissementHistorique_utf8.csv", 
                dtypes=dtypes)
    other_df=pl.read_database(your_query, your_conn_str).with_columns(pl.col(string_cols_to_join_on).cast(pl.Categorical())).lazy()
    

As an aside, some quick stats...

From here if I do df.collect().estimated_size('gb') it comes back as 4.57.

In contrast if I do pl.read_csv(thefile).estimated_size('gb') then it's 10.5 so there are significant savings in changing the column types.

You might need to keep everything in that StringCache context until you collect, I'm not sure how the StringCache interacts with LazyFrames. Since it takes basically no time for me to run the scan_csv, it's clearly not establishing the Categorical indexing right away.

It might also behoove you to resave the csv as multiple parquets that you then lazily load as a dataset via pyarrow.

polars.scan_csv produces a query plan (called a LazyFrame). You can then build you query and on the end call collect to materialize a DataFrame.

This is the case for all scan_ methods. The benefit of this is that the Polars optimizer then can push down optimizations into the readers. It can apply filters in the readers and only selects columns it needs. This can save a lot of work.

polars.read_csv can be seen as polars.scan_csv().collect(). E.g. you simply read all the data and immediately produce a DataFrame. This means that you might do work that was not needed. The Polars optimizer is not able to do anything if you want a result immediately.

I don't agree with the other answer that polars.read_csv only should be used when data is large. It is just as well suitable for smaller data.

You've found a known bug

Here's a workaround class to use in place of pl.read_csv_batched until that's fixed.

from math import ceil
import polars as pl
class BatchedPolarsCSV:
    def __init__(self, source: str, batch_size: int, **kwargs):
        if "batch_size" in kwargs:
            del kwargs["batch_size"]
        self._reader = pl.read_csv_batched(source, **kwargs)
        self._batch_size = batch_size
        self._native_batch_size = None
        self._cached_rows = []
        self._exhausted = False
    
    def _batches_to_get(self, target_rows:int)->int:
        need_rows = target_rows - self._have_rows()
        if self._native_batch_size:
            return ceil(need_rows/self._native_batch_size)
        else:
            return 1
    def _have_rows(self):
        return sum([x.height for x in self._cached_rows])
    
    def next_batches(self, n: int) -> None | list[pl.DataFrame]:
        if self._exhausted:
            return None
        target_rows = n * self._batch_size
        while self._have_rows() < target_rows:
            chunk = self._reader.next_batches(self._batches_to_get(target_rows))
            if chunk is None:
                break
            self._native_batch_size = chunk[0].height
            self._cached_rows.extend(chunk)
        if len(self._cached_rows)==0:
            self._exhausted=True
            return None
        df = pl.concat(self._cached_rows)
        return_part = df.slice(0, n * self._batch_size)
        cache_part = df.slice(n * self._batch_size)
        if cache_part.height > 0:
            self._cached_rows = [cache_part]
        else:
            self._cached_rows = []
        return return_part.with_columns(
            (pl.int_range(pl.len()) / self._batch_size)
            .floor()
            .alias(PARTITION_COL_NAME := "__parition_by_col")
        ).partition_by(PARTITION_COL_NAME, include_key=False)

Have you tried increasing the memory limits for your Cloud Run service? If your memory is 8GB or more, the minimum required CPUs are 4 vCPU. You could try increasing it to see if it works; if not, the issue might be related to how Polars operates in the background. For out-of-memory errors in Cloud Run, you can review this documentation to see if it applies to your situation.

If the above doesn't work and the issue is not related to Cloud Run. You may also be affected (your polar version) by a recent announcement from the Polars team regarding the new streaming engine (#20947).

The old streaming engine is being deprecated, and the new streaming engine is coming to replace it soon™. The new streaming engine is far along, but does not yet have 100% feature parity with the old streaming engine. At the same time, the maintenance burden of maintaining both the old and new streaming engine is becoming too large. Therefore, we are deprecating the old streaming engine starting on Polars version 1.23.0 and telling people to pin their Polars versions to any version before 1.23.0.

You can also verify whether you're really affected by opening an issue on GitHub. The Polars team seems actively responding to queries and may provide workarounds.