Your first example works because a Series has a multiplication method.

For example if you do

Copyfunc(pl.Series([1,2,3,4,5]))

then you get back a series of the original multiplied by 2.

Your func2 is just an anonymous function. To use map_batches, your function needs to operate on the entire column and return something like a Series.

For instance:

Copyfrom lxml import etree as ET
def func2_series(xml_strings):
    ret_List=[]
    for xml_string in xml_strings:
        root = ET.fromstring(xml_string, ET.XMLParser(recover=True))
        text_list = []
        for elem in root.iter():
            text = elem.text.strip() if elem.text else ''
            text_list.append(text)
        ret_List.append(text_list)
    return pl.Series(ret_List)

followed by

Copydf.with_columns(pl.col("B").map_batches(func2_series).alias('new_col2'))

will work.

Alternatively if you have

Copydef func2(xml_string):
    root = ET.fromstring(xml_string, ET.XMLParser(recover=True))
    text_list = []
    for elem in root.iter():
        text = elem.text.strip() if elem.text else ''
        text_list.append(text)
    return text_list

then you can use map_elements and polars will do the looping for you.

Copydfpl.with_columns(pl.col("B").map_elements(func2))

btw, you don't need to use a lambda if the function you're passing accepts the exact x that you have. In other words where you have .map_batches(lambda x: func2(x)) you can just do .map_batches(func2). The lambda comes into play if you need to transform the parameters.

The functionality is available natively in Polars via the .str namespace.

.str.split() doesn't support regex.

But similar behaviour can be achieved with .extract_all() and .replace_all()

df = pl.DataFrame({"content": ["o neHItw oHIIIIIth ree", "fo urHIIfi veHIIIIs ix"]})

pattern2 = r"HI+"
pattern3 = r"\s"

replacement = ""

df.with_columns(
   pl.col("content").str.extract_all(rf".*?({pattern2}|$)")
     .alias("sentences")
)

shape: (2, 2)
┌────────────────────────┬────────────────────────────────────┐
│ content                ┆ sentences                          │
│ ---                    ┆ ---                                │
│ str                    ┆ list[str]                          │
╞════════════════════════╪════════════════════════════════════╡
│ o neHItw oHIIIIIth ree ┆ ["o neHI", "tw oHIIIII", "th ree"] │
│ fo urHIIfi veHIIIIs ix ┆ ["fo urHII", "fi veHIIII", "s ix"] │
└────────────────────────┴────────────────────────────────────┘

list.eval() could then be used to process the list and "extract" the desired result.

df.with_columns(
   pl.col("content").str.extract_all(rf".*?({pattern2}|$)")
     .list.eval(
        pl.element().str.replace_all(pattern2, "")
                    .str.replace_all(pattern3, replacement)
     )
     .alias("normal_text")
)

shape: (2, 2)
┌────────────────────────┬─────────────────────────┐
│ content                ┆ normal_text             │
│ ---                    ┆ ---                     │
│ str                    ┆ list[str]               │
╞════════════════════════╪═════════════════════════╡
│ o neHItw oHIIIIIth ree ┆ ["one", "two", "three"] │
│ fo urHIIfi veHIIIIs ix ┆ ["four", "five", "six"] │
└────────────────────────┴─────────────────────────┘

Performance

A basic comparison of both approaches.

N = 2000
df = pl.DataFrame({
   "content": [
      "o neHItw oHIIIIIth ree" * N, 
      "fo urHIIfi veHIIIIs ix" * N] * N
})

The idea is to use Polars Expressions instead of applying custom Python functions/lambdas.

It looks like you're trying to count when ref and another column have the same value?

df.select(pl.exclude("ref") == pl.col("ref"))

shape: (3, 2)
┌───────┬───────┐
│ v1    ┆ v2    │
│ ---   ┆ ---   │
│ bool  ┆ bool  │
╞═══════╪═══════╡
│ true  ┆ true  │
│ false ┆ false │
│ false ┆ true  │
└───────┴───────┘

.sum_horizontal() can be used to get a "count" of the true values on each row.

df.with_columns(count = pl.sum_horizontal(pl.exclude("ref") == pl.col("ref")))

shape: (3, 4)
┌─────┬─────┬─────┬───────┐
│ ref ┆ v1  ┆ v2  ┆ count │
│ --- ┆ --- ┆ --- ┆ ---   │
│ i64 ┆ i64 ┆ i64 ┆ u32   │
╞═════╪═════╪═════╪═══════╡
│ -1  ┆ -1  ┆ -1  ┆ 2     │
│ 2   ┆ 5   ┆ 5   ┆ 0     │
│ 8   ┆ 0   ┆ 8   ┆ 1     │
└─────┴─────┴─────┴───────┘

You can use the expression syntax to select all columns with pl.all() and then map_batches the numpy np.log2(..) function over the columns.

df.select(
    pl.all().map_batches(np.log2)
)

Note that we choose map_batches here as map_elements would call the function upon each value.

map_elements = pl.Series(np.log2(value) for value in pl.Series([1, 2, 3]))

But np.log2 can be called once with multiple values, which would be faster.

map_batches = np.log2(pl.Series([1, 2, 3]))

See the User guide for more.

• map_elements: Call a function separately on each value in the Series.
• map_batches: Always passes the full Series to the function.

Numpy

Polars expressions also support numpy universal functions.

That means you can pass a polars expression to a numpy ufunc:

df.select(
    np.log2(pl.all())
)

Use replace_strict:

In [21]: data1 = {"a": [1, 2, 3, 4], "b1": [11, 12, 13, 14], "c1" : [31, 32, 33, 34]}
    ...: df1_pl = pl.DataFrame(data1)
    ...: print(df1_pl)
    ...: weekday = ['Monday', 'Tuesday', 'Wednesday', 'Thursday']
    ...:
    ...: print(df1_pl.with_columns(
    ...: weekday=pl.col('a').replace_strict({idx: val for idx, val in enumerate(weekday, start=1)})
    ...: ))
shape: (4, 3)
┌─────┬─────┬─────┐
│ a   ┆ b1  ┆ c1  │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1   ┆ 11  ┆ 31  │
│ 2   ┆ 12  ┆ 32  │
│ 3   ┆ 13  ┆ 33  │
│ 4   ┆ 14  ┆ 34  │
└─────┴─────┴─────┘
shape: (4, 4)
┌─────┬─────┬─────┬───────────┐
│ a   ┆ b1  ┆ c1  ┆ weekday   │
│ --- ┆ --- ┆ --- ┆ ---       │
│ i64 ┆ i64 ┆ i64 ┆ str       │
╞═════╪═════╪═════╪═══════════╡
│ 1   ┆ 11  ┆ 31  ┆ Monday    │
│ 2   ┆ 12  ┆ 32  ┆ Tuesday   │
│ 3   ┆ 13  ┆ 33  ┆ Wednesday │
│ 4   ┆ 14  ┆ 34  ┆ Thursday  │
└─────┴─────┴─────┴───────────┘