Check your DataFrame with data.columns

It should print something like this

Index([u'regiment', u'company',  u'name',u'postTestScore'], dtype='object')

Check for hidden white spaces..Then you can rename with

data = data.rename(columns={'Number ': 'Number'})

Two things to fix:

First, when you apply a lambda function to a pandas Series, the lambda function is applied to each element of the Series. What I think you need is to apply your function to the entire Series in a vectorized manner.

Second, your function has multiple return statements. As a result, only the first statement, return source.replace('[^A-Za-z]',' '), will ever run. What you need to do is make your preprocessing changes on the variable source inside your function, and finally return the modified source (or an intermediate variable) at the very end.

To rewrite your function to operate on an entire pandas Series, replace every occurrence of source. with source.str.. The new function definition:

def data_preprocessing(source):
    source = source.str.replace('[^A-Za-z]',' ')
    #data['description'] = data['description'].str.replace('\W+',' ')
    source = source.str.lower()
    source = source.str.replace("\s\s+" , " ")
    source = source.str.replace('\s+[a-z]{1,2}(?!\S)',' ')
    source = source.str.replace("\s\s+" , " ")
    return source

Then, instead of this:

data['description'] = data['description'].apply(lambda row: data_preprocessing(row))
data['event_name'] = data['event_name'].apply(lambda row: data_preprocessing(row))

Try this:

data['description'] = data_preprocessing(data['description'])
data['event_name'] = data_preprocessing(data['event_name'])

As of pandas 2.0, append (previously deprecated) was removed.

You need to use concat instead (for most applications):

df = pd.concat([df, pd.DataFrame([new_row])], ignore_index=True)

As noted by @cottontail, it's also possible to use loc, although this only works if the new index is not already present in the DataFrame (typically, this will be the case if the index is a RangeIndex:

df.loc[len(df)] = new_row # only use with a RangeIndex!

Why was it removed?

We frequently see new users of pandas try to code like they would do it in pure Python. They use iterrows to access items in a loop (see here why you shouldn't), or append in a way that is similar to python list.append.

However, as noted in pandas' issue #35407, pandas's append and list.append are really not the same thing. list.append is in place, while pandas's append creates a new DataFrame:

I think that we should deprecate Series.append and DataFrame.append. They're making an analogy to list.append, but it's a poor analogy since the behavior isn't (and can't be) in place. The data for the index and values needs to be copied to create the result.

These are also apparently popular methods. DataFrame.append is around the 10th most visited page in our API docs.

Unless I'm mistaken, users are always better off building up a list of values and passing them to the constructor, or building up a list of NDFrames followed by a single concat.

As a consequence, while list.append is amortized O(1) at each step of the loop, pandas' append is O(n), making it inefficient when repeated insertion is performed.

What if I need to repeat the process?

Using append or concat repeatedly is not a good idea (this has a quadratic behavior as it creates a new DataFrame for each step).

In such case, the new items should be collected in a list, and at the end of the loop converted to DataFrame and eventually concatenated to the original DataFrame.

lst = []

for new_row in items_generation_logic:
    lst.append(new_row)

# create extension
df_extended = pd.DataFrame(lst, columns=['A', 'B', 'C'])
# or columns=df.columns if identical columns

# concatenate to original
out = pd.concat([df, df_extended])

Double check if there's a space in the column name. 'Survived ' vs 'Survived' It happens more often than you'd think especially with CSV data source.

Try:

result.columns = ["_id", "IdList", "levels.active", "levels.level", "levels.actions.isActive"]

Or a "dynamic" version as per your second question:

result.columns = [col.replace("_0", "") for col in result.columns]

"sklearn.datasets" is a scikit package, where it contains a method load_iris().

load_iris(), by default return an object which holds data, target and other members in it. In order to get actual values you have to read the data and target content itself.

Whereas 'iris.csv', holds feature and target together.

FYI: If you set return_X_y as True in load_iris(), then you will directly get features and target.

from sklearn import datasets
data,target = datasets.load_iris(return_X_y=True)

That way of merging is creating a Pandas DataFrame. Try this:

import geopandas as gpd

df = gpd.read_file(r"/home/bera/Drives/data1/data/LMV/LMV-data_2021-10/ok_riks_Sweref_99_TM_shape/oversikt/riks/ak_riks.shp")

gdf1 = df.head(10)
gdf2 = df.tail(10)

print(type(gdf1.merge(gdf2, on="LANSKOD", how="outer"))) #This is creating a pandas df
#pandas.core.frame.DataFrame

merged = gpd.pd.merge(gdf1, gdf2, on="LANSKOD", how="outer")
print(type(merged))
#<class 'geopandas.geodataframe.GeoDataFrame'> #Now it is a geopandas df

merged["geometry"] = merged["geometry_x"].combine_first(merged["geometry_y"]) #Create a complete geometry column
merged = merged.drop(columns=["geometry_x", "geometry_y"])
merged.to_file(r"/home/bera/Desktop/GIStest/testmerge.shp")