This will definitely slow things down for a large Series, but you can pass a ternary expression with a callable:

>>> b.apply(lambda x: x.decode('utf-8') if isinstance(x, bytes) else x)                                                                                                                                                                                      
0       123
1      434,
2       fgd
3       aas
4    442321
dtype: object

Looking at the source for .str.decode() is instructive - it just applies _na_map(f, arr) over the Series, where the function f is f = lambda x: x.decode(encoding, errors). Because str doesn't have a "decode" method to begin with, that error will become NaN. This happens in str_decode().

>>> from pandas.core.strings import str_decode                                                                                                                                                                                                               
>>> from pandas.core.strings import _cpython_optimized_encoders                                                                                                                                                                                              

>>> "utf-8" in _cpython_optimized_encoders                                                                                                                                                                                                                   
True
>>> str_decode(b, "utf-8")                                                                                                                                                                                                                                   
array([nan, nan, nan, nan, '442321'], dtype=object)

>>> from pandas.core.strings import _na_map                                                                                                                                                                                                                  
>>> f = lambda x: x.decode("utf-8")                                                                                                                                                                                                                          
>>> _na_map(f, b)                                                                                                                                                                                                                                            
array([nan, nan, nan, nan, '442321'], dtype=object)

You can use this solution implemented to pandas by Series.apply:

from Crypto.Cipher import XOR
import base64

def encrypt(key, plaintext):
  cipher = XOR.new(key)
  return base64.b64encode(cipher.encrypt(plaintext))

def decrypt(key, ciphertext):
  cipher = XOR.new(key)
  return cipher.decrypt(base64.b64decode(ciphertext))

load['Encoded_Column'] = load['F'].apply(lambda x: encrypt('password',x))
load['Decoded_Column'] = (load['Encoded_Column'].apply(lambda x: decrypt('password', x))
                                                .str.decode("utf-8"))
print (load)
   A  B  C  D  E  F Encoded_Column Decoded_Column
0  a  4  7  1  5  a        b'EQ=='              a
1  b  5  8  3  3  a        b'EQ=='              a
2  c  4  9  5  6  a        b'EQ=='              a
3  d  5  4  4  9  b        b'Eg=='              b
4  e  5  2  2  2  b        b'Eg=='              b
5  f  4  0  0  4  b        b'Eg=='              b

Another solution:

import base64
def encode(key, clear):
    enc = []
    for i in range(len(clear)):
        key_c = key[i % len(key)]
        enc_c = chr((ord(clear[i]) + ord(key_c)) % 256)
        enc.append(enc_c)
    return base64.urlsafe_b64encode("".join(enc).encode()).decode()

def decode(key, enc):
    dec = []
    enc = base64.urlsafe_b64decode(enc).decode()
    for i in range(len(enc)):
        key_c = key[i % len(key)]
        dec_c = chr((256 + ord(enc[i]) - ord(key_c)) % 256)
        dec.append(dec_c)
    return "".join(dec)

load['Encoded_Column'] = load['F'].apply(lambda x: encode('password',x))
load['Decoded_Column'] = load['Encoded_Column'].apply(lambda x: decode('password', x))

Or use list comprehension:

load['Encoded_Column'] = [encode('password',x) for x in load['F']]
load['Decoded_Column'] = [decode('password', x) for x in load['Encoded_Column']]

print (load)
   A  B  C  D  E  F Encoded_Column Decoded_Column
0  a  4  7  1  5  a           w5E=              a
1  b  5  8  3  3  a           w5E=              a
2  c  4  9  5  6  a           w5E=              a
3  d  5  4  4  9  b           w5I=              b
4  e  5  2  2  2  b           w5I=              b
5  f  4  0  0  4  b           w5I=              b

You can look at my answer here to know the proper usage of LabelEncoder for multiple columns:-

Why does sklearn preprocessing LabelEncoder inverse_transform apply from only one column?

The explanation is that LabelEncoder only supports single dimension as input. So for each column, you need to have a different labelEncoder object which can then be used to inverse transform that particular column only.

You can use a dictionary of labelencoder objects for convertig multiple columns. Something like this:

labelencoder_dict = {}
for col in b.columns:
    labelEncoder = preprocessing.LabelEncoder()
    b[col] = labelEncoder.fit_transform(b[col])
    labelencoder_dict[col]=labelEncoder

While decoding, you can just use:

for col in b.columns:
    b[col] = labelencoder_dict[col].inverse_transform(b[col])

Update:-

Now that you have added the column which you are using as y, here's how you can decode it (assuming you have added the 'Predicted_Values' column to the dataframe):

for col in b.columns:
    # Skip the predicted column here
    if col != 'Predicted_valu‌​es':
        b[col] = labelencoder_dict[col].inverse_transform(b[col])

# Use the original `y (Activity_Profile)` encoder on predicted data
b['Predicted_valu‌​es'] = labelencoder_dict['Activity_Profile'].inverse_transfo‌​rm(
                                                      b['Predicted_valu‌​es'])