Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so slow. In the source code of SimpleImputer there is also the comment that explains why they do not use the scipy.stats.mstats.mode, which is mush faster:

scipy.stats.mstats.mode cannot be used because it will no work properly if the first element is masked and if its frequency is equal to the frequency of the most frequent valid element. See https://github.com/scipy/scipy/issues/2636

So if you want to use the SimpleImputer with this strategy, a faster way would be to use the "constant" strategy and pass the most frequent value by yourself (ser.mode()[0]) then the time is almost the same:

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imp.fit_transform(arr)
print('Simple Imputer (Most Frequent) Time Elapsed:', time.time()-t0)

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=ser.mode()[0])
imp.fit_transform(arr)
print('Simple Imputer (Constant) Time Elapsed:', time.time()-t0)

t0 = time.time()
ser.fillna(value=ser.mode()[0])
print('Pandas Time Elapsed:', time.time()-t0)

And the time elapsed for each strategy:

Simple Imputer (Most Frequent) Time Elapsed: 14.320188045501709
Simple Imputer (Constant) Time Elapsed: 0.052472829818725586
Pandas Time Elapsed: 0.04726815223693848

UPDATE:

Based on feedback from OP, the strategy that gives the desired result is to do this:

X_train[X_train_objects] = imputer.fit_transform(X_train[X_train_objects].values)

ORIGINAL ANSWER:

Here's what the code in your question does:

• works with X_train[X_train_objects], which has shape (5, 6)
• converts it to a numpy array (via values) and changes it to a 1D array of length 30 using .reshape(-1,1)[:,0]
• passes this as an argument to imputer.fit_transform which returns a result whose shape is the same as its input
• attempts to use this 1D array of length 30 to update all rows in X_train[X_train_objects] which (as mentioned above) has shape (5, 6), or specifically, has only 6 columns

This gives rise to the error: ValueError: Columns must be same length as key

What I believe you intend is, having massaged the values originally found in X_train[X_train_objects], to update the original object by overwriting the original values with the massaged ones. To do this, I think the following should work:

X_train[X_train_objects] = (
    imputer.fit_transform(X_train[X_train_objects].values.reshape(-1,1))[:,0]
    .reshape(-1,len(X_train_objects)) )

To use mean values for numeric columns and the most frequent value for non-numeric columns you could do something like this. You could further distinguish between integers and floats. I guess it might make sense to use the median for integer columns instead.

import pandas as pd
import numpy as np

from sklearn.base import TransformerMixin

class DataFrameImputer(TransformerMixin):

    def __init__(self):
        """Impute missing values.

        Columns of dtype object are imputed with the most frequent value 
        in column.

        Columns of other types are imputed with mean of column.

        """
    def fit(self, X, y=None):

        self.fill = pd.Series([X[c].value_counts().index[0]
            if X[c].dtype == np.dtype('O') else X[c].mean() for c in X],
            index=X.columns)

        return self

    def transform(self, X, y=None):
        return X.fillna(self.fill)

data = [
    ['a', 1, 2],
    ['b', 1, 1],
    ['b', 2, 2],
    [np.nan, np.nan, np.nan]
]

X = pd.DataFrame(data)
xt = DataFrameImputer().fit_transform(X)

print('before...')
print(X)
print('after...')
print(xt)

which prints,

before...
     0   1   2
0    a   1   2
1    b   1   1
2    b   2   2
3  NaN NaN NaN
after...
   0         1         2
0  a  1.000000  2.000000
1  b  1.000000  1.000000
2  b  2.000000  2.000000
3  b  1.333333  1.666667