1. Decode the string to Unicode. Assuming it's UTF-8-encoded:

   str.decode("utf-8")
   

2. Call the replace method and be sure to pass it a Unicode string as its first argument:

   str.decode("utf-8").replace(u"\u2022", "*")
   

3. Encode back to UTF-8, if needed:

   str.decode("utf-8").replace(u"\u2022", "*").encode("utf-8")
   

(Fortunately, Python 3 puts a stop to this mess. Step 3 should really only be performed just prior to I/O. Also, mind you that calling a string str shadows the built-in type str.)

GetField returns UTF-8 encoded strings and you'll want to decode it before you process it in any way. Then you encode the result to pass it to SetField. You've got it backwards.

Fiona (shameless plug) deals in Python unicode strings and so is simpler to use.

Unidecode (https://pypi.python.org/pypi/Unidecode) is handy for stuff like this because it will make sensible transliterations and romanizations for many languages. It looks like it would make the ones you want.

>>> from unidecode import unidecode
>>> unidecode(u'\u00c2')
'A'
>>> unidecode(u'\u00C9')
'E'
>>> unidecode(u'\u00C8')
'E'

The example below uses Natural Earth data and converts "Côte d'Ivoire" to "Cote d'Ivoire", etc, without presuming anything about the characters in the source data.

import fiona
from unidecode import unidecode

with fiona.open(
        '/Users/seang/data/ne_50m_admin_0_countries/'
        'ne_50m_admin_0_countries.shp', 'r') as source:

    # Create an output shapefile with the same schema,
    # coordinate systems. ISO-8859-1 encoding.
    with fiona.open(
            '/tmp/transliterated.shp', 'w',
            **source.meta) as sink:

        # Identify all the str type properties.
        str_prop_keys = [
            k for k, v in sink.schema['properties'].items()
                if v.startswith('str')]

        for rec in source:

            # Transliterate and update each of the str properties.
            for key in str_prop_keys:
                val = rec['properties'][key]
                if val:
                    rec['properties'][key] = unidecode(val)

            # Write out the transformed record.
            sink.write(rec)

The string you're searching for to replace must also be a Unicode string. Try:

newToke = thisToken.replace(u'\u2013','')

If you have a bytestring (undecoded data), use the 'replace' error handler. For example, if your data is (mostly) UTF-8 encoded, then you could use:

decoded_unicode = bytestring.decode('utf-8', 'replace')

and U+FFFD � REPLACEMENT CHARACTER characters will be inserted for any bytes that can't be decoded.

If you wanted to use a different replacement character, it is easy enough to replace these afterwards:

decoded_unicode = decoded_unicode.replace('\ufffd', '#')

Demo:

>>> bytestring = b'F\xc3\xb8\xc3\xb6\xbbB\xc3\xa5r'
>>> bytestring.decode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbb in position 5: invalid start byte
>>> bytestring.decode('utf8', 'replace')
'Føö�Bår'

In Python 3, this works (although the print may not, depending on your terminal):

>>> s="bla bla bla \uf604 bla bla bla"
>>> print(s)
bla bla bla  bla bla bla
>>> s="bla bla bla \uf604 bla bla bla"
>>> s.replace('\uf604','X')
'bla bla bla X bla bla bla'

But perhaps you have a literal slash and not an escape code. Note the print difference:

>>> s="bla bla bla \\uf604 bla bla bla"
>>> print(s)
bla bla bla \uf604 bla bla bla
>>> s.replace('\uf604','X')
'bla bla bla \\uf604 bla bla bla'

Use a escape slash to fix:

>>> s.replace('\\uf604','X')
'bla bla bla X bla bla bla'

There seem to be two problems with your program.

Firstly, you are passing the wrong code point to chr(). The hexdecimal code point of the character ’ is 0x2019, but you are passing in the decimal number 2019 (which equates to 0x7e3 in hexadecimal). So you need to do either:

    temp = temp.replace(chr(0x2019), "'") # hexadecimal

or:

    temp = temp.replace(chr(8217), "'") # decimal

in order to replace the character correctly.

Secondly, the reason you are getting the error is because some other part of your program (probably the database backend) is trying to encode unicode strings using some encoding other than UTF-8. It's hard to be more precise about this, because you did not include the full traceback in your question. However, the reference to "charmap" suggests a Windows code page is being used (but not cp1252); or an iso encoding (but not iso8859-1, aka latin1); or possibly KOI8_R.

Anyway, the correct way to deal with this issue is to ensure all parts of your program (and especially the database) use UTF-8. If you do that, you won't have to mess about replacing characters anymore.

You need a regex substitution:

re.sub(r'[^\u0020-\ud7ff]', '', s)

where s is the input string.

Code:

import re

s = " you�"
print(re.sub(r'[^\u0020-\ud7ff]', '', s))
#  you

I believe the problem is that Vim not opening the file using the right encoding. Your file is in encoding cp1252 but Vim doesn't detect it and guess utf-8.

I would propose to open the file with the ++enc=cp1252 flag.

:e ++enc=cp1252 filepath

To improve the way Vim guess the encoding you can add the following lines (Vim would first try to open as cp1252 and then as utf-8):

:set fileencodings=cp1252,utf-8

utf-8 encoding can represent the character ü; there is no error, so no replacement occurs.

To see errors='replace' in action, use another encoding that cannot represent the character. For example ascii:

>>> 'Brückenspinne'.encode("ascii", errors='replace')
b'Br?ckenspinne'

>>> 'Brückenspinne'.encode("ascii", errors='xmlcharrefreplace')
b'Brückenspinne'