I spot my error, I was switching the instantiation of the classes: the custom transformers have to be instantiated inside the ColumnTransformer, while the ColumnTransformer does not have to be instantiated inside the pipeline.

The correct code is the following:

transformation_pipeline = ColumnTransformer([
    ('adoption', TransformAdoptionFeatures(), features_adoption),
    ('census', TransformCensusFeaturesRegr(), features_census),
    ('climate', TransformClimateFeatures(), features_climate),
    ('soil', TransformSoilFeatures(), features_soil),
    ('economic', TransformEconomicFeatures(), features_economic)
],
    remainder='drop')

full_pipeline_stand = Pipeline([
    ('transformation', transformation_pipeline),
    ('scaling', StandardScaler())
])

Okay this seems to kinda work, but I still would like to have better column names:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import FunctionTransformer

def take_cube(data, col):
    data["speed_cube"] = np.power(data[col], 3)
    return data[['speed_cube']]

def sin_angle(data, speed, acc):
    data["angle"] = data[speed] * np.sin(data[acc])
    return data[['angle']]

preprocessor = ColumnTransformer(
    transformers=[
        ("step1", FunctionTransformer(take_cube, kw_args={'col': 'speed'}, validate=False), ['speed']),
        ("step2", FunctionTransformer(sin_angle, kw_args={'speed': 'speed', 'acc': 'acceleration'}, validate=False), ['speed', 'acceleration']),
    ],
    remainder="passthrough"
).set_output(transform="pandas")
new_data = preprocessor.fit_transform(data)

print(new_data)

Output:

                   step1__speed_cube  step2__angle  remainder__heart-rate  \
                                                                              
2020-08-18 14:43:19          80.901828      0.380109                  102.0   
2020-08-18 14:43:20          81.520685      0.364660                  103.0   
2020-08-18 14:43:21          85.707790      0.103161                  105.0   
2020-08-18 14:43:22          87.824421      0.007112                  106.0   
2020-08-18 14:43:23          87.587538      0.506943                  106.0   
...                                ...           ...                    ...   
2020-09-13 14:55:57           1.170905      0.024661                  130.0   
2020-09-13 14:55:58           0.569723      0.021386                  130.0   
2020-09-13 14:55:59           0.233745     -0.103366                  129.0   
2020-09-13 14:56:00           0.000000     -0.000000                  130.0   
2020-09-13 14:56:01           0.000000     -0.000000                  130.0   

                     remainder__cadence  remainder__slope  remainder__angle  
                                                                             
2020-08-18 14:43:19                64.0         -0.033870         -0.033857  
2020-08-18 14:43:20                64.0         -0.033571         -0.033559  
2020-08-18 14:43:21                66.0         -0.033223         -0.033210  
2020-08-18 14:43:22                66.0         -0.032908         -0.032896  
2020-08-18 14:43:23                67.0          0.000000          0.000000  
...                                 ...               ...               ...  
2020-09-13 14:55:57                 0.0          0.000000          0.000000  
2020-09-13 14:55:58                 0.0          0.000000          0.000000  
2020-09-13 14:55:59                 0.0          0.000000          0.000000  
2020-09-13 14:56:00                 0.0          0.000000          0.000000  
2020-09-13 14:56:01                 0.0          0.000000          0.000000  

[38254 rows x 6 columns]```

Notice that I had to call `step1` and `step2` on the ColumnTransformer otherwise the naming of the column would look something like `cube__speed_cube` for the first step...

There is mismatch between the columns that the function transformer returns, and its feature_names_out= name. They are clashing, both in terms of how many features are returned and their name. I firstly made sure that each function transformer only returns the one column (or more) that it's supposed to, and also changed the names to make them consistent. This way, the number of features and names defined in feature_names_out= matches the actual returned data for each transformer.

A second issue was that new features are created in the ColumnTransformer, and an attempt is made to access those variables by other parts of the same ColumnTransformer. This won't work as the column transformer runs things in parallel. What you'd need to do is chain the column transformers in a sequence, such that the next column transformer can access the new variables created from the earlier one. I have made this change to the code.

I made some other minor changes, including forcing transformers to return pandas dataframes rather than numpy arrays.

The new preprocessor:

Its output column names:

Index(['Sex_female', 'Sex_male', 'Embarked_C', 'Embarked_Q', 'Embarked_S',
       'traveling_category_A', 'traveling_category_B', 'traveling_category_C',
       'age_interval_(0, 10]', 'age_interval_(10, 20]',
       'age_interval_(20, 30]', 'age_interval_(30, 40]',
       'age_interval_(40, 50]', 'age_interval_(50, 60]',
       'age_interval_(60, 70]', 'age_interval_(70, 100]', 'Pclass', 'Fare',
       'PassengerId', 'Survived', 'Name', 'Ticket', 'Cabin'],
      dtype='object')

Code:

import pandas as pd
titanic_data = pd.read_csv('../titanic.csv')

from sklearn.pipeline import make_pipeline, FunctionTransformer
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer

from sklearn import set_config
set_config(transform_output='pandas')

#
# Sum relatives
#
def sum_name(function_transformer, feature_names_in):
    return ["total_relatives"]  # feature names out

def sum_relatives(X):
    X_copy = X.copy()
    X_copy['total_relatives'] = X_copy['SibSp'] + X_copy['Parch']
    return X_copy[['total_relatives']]

total_relatives_pipeline = make_pipeline(
    FunctionTransformer(sum_relatives, feature_names_out=sum_name)
)

#
#Categorize travel
#
def cat_travel_name(function_transformer, feature_names_in):
    return ["traveling_category"]  # feature names out

def categorize_travel(X):
    X_copy = X.copy()

    conditions = [
        (X_copy['total_relatives'] == 0),
        (X_copy['total_relatives'] >= 1) & (X_copy['total_relatives'] <= 3),
        (X_copy['total_relatives'] >= 4)
    ]
    categories = ['A', 'B', 'C']

    X_copy['traveling_category'] = np.select(conditions, categories, default='Unknown')
    return X_copy[['traveling_category']]

travel_category_pipeline = make_pipeline(
    FunctionTransformer(categorize_travel, feature_names_out=cat_travel_name)
)

#
# Ordinal encoder
#
class_order = [[1, 2, 3]]

ord_pipeline = make_pipeline(
    OrdinalEncoder(categories=class_order)    
    )

cat_pipeline = make_pipeline(
    SimpleImputer(strategy="most_frequent"),
    OneHotEncoder(handle_unknown="ignore", sparse_output=False)
    )

#Numerical for fare
fare_pipeline = make_pipeline(
        StandardScaler()
    )

#
# Age transformer
#
def interval_name(function_transformer, feature_names_in):
    return ["age_interval"]  # feature names out

def age_transformer(X):
    X_copy = X.copy()
    median_age_by_class = X_copy.groupby('Pclass')['Age'].median().reset_index()
    median_age_by_class.columns = ['Pclass', 'median_age']
    for index, row in median_age_by_class.iterrows():
        class_value = row['Pclass']
        median_age = row['median_age']
        X_copy.loc[X_copy['Pclass'] == class_value, 'Age'] = \
            X_copy.loc[X_copy['Pclass'] == class_value, 'Age'].fillna(median_age)
    bins = [0, 10, 20, 30, 40, 50, 60, 70, 100]
    X_copy['age_interval'] = pd.cut(X_copy['Age'], bins=bins)
    return X_copy[['age_interval']]

def age_processor():
    return make_pipeline(
        FunctionTransformer(age_transformer, feature_names_out=interval_name),
        )

#
# Column transformers
#
preprocessing_initial = ColumnTransformer([
        ("ord", ord_pipeline, ['Pclass']),
        ("age_processing", age_processor(), ['Pclass', 'Age']),
        ("num", fare_pipeline, ['Fare']),
        ("total_relatives", total_relatives_pipeline, ['SibSp', 'Parch'])],
        remainder='passthrough',
        verbose_feature_names_out=False
)

preprocessing_travel_category = ColumnTransformer(
    [("travel_category", travel_category_pipeline, ['total_relatives'])],
    remainder='passthrough',
    verbose_feature_names_out=False
)

preprocessing_cat = ColumnTransformer(
    [("cat", cat_pipeline, ['Sex', 'Embarked', 'traveling_category', 'age_interval'])],
    remainder='passthrough',
    verbose_feature_names_out=False
)

#Final transformer
preprocessor = make_pipeline(
    preprocessing_initial,
    preprocessing_travel_category,
    preprocessing_cat
)

#Run
preprocessor.fit(titanic_data)
preprocessor.fit_transform(titanic_data).columns