Try setting LANG=C and then run it over the Unicode range:
echo "hi ☠ there ☠" | LANG=C sed "s/[\x80-\xFF]//g"

• Use the fileinput module to loop over standard input or a list of files,
• decode the lines you read from UTF-8 to unicode objects
• then map any unicode characters you desire with the translate method

translit.py would look like this:

#!/usr/bin/env python2.6
# -*- coding: utf-8 -*-

import fileinput

table = {
          0xe4: u'ae',
          ord(u'ö'): u'oe',
          ord(u'ü'): u'ue',
          ord(u'ß'): None,
        }

for line in fileinput.input():
    s = line.decode('utf8')
    print s.translate(table), 

And you could use it like this:

$ cat utf8.txt 
sömé täßt
sömé täßt
sömé täßt

$ ./translit.py utf8.txt 
soemé taet
soemé taet
soemé taet

• Update:

In case you are using python 3 strings are by default unicode and you dont' need to encode it if it contains non-ASCII characters or even a non-Latin characters. So the solution will look as follow:

line = 'Verhältnismäßigkeit, Möglichkeit'

table = {
         ord('ä'): 'ae',
         ord('ö'): 'oe',
         ord('ü'): 'ue',
         ord('ß'): 'ss',
       }

line.translate(table)

>>> 'Verhaeltnismaessigkeit, Moeglichkeit'

Encode string as unicode.

>>> special = u"\u2022"
>>> abc = u'ABC•def'
>>> abc.replace(special,'X')
u'ABCXdef'

I would propose you:

:s/\([\x80-\xFF]\)/\='0x'.printf('%02x', char2nr(submatch(1)))/g

Hi Jay,

Thank you very much - the macros in the article worked just great!

This was the first time I ever dabbled with VBA in Word, and this worked almost first time. (I do have quite a lot of experience writing VBA macros in Excel, but never in Word.)

Something I learned from using these macros for my specific case - to find and replace all the ?-in-a-rectangles with a regular double quote - is that, in Word t,here is a difference between an opening double quote mark and a closing double quote mark.

At first I highlighted a double quote mark in my document (by chance, a closing one), and used the code that the first macro returned to replace ALL of the ?-in-a-rectangles. This caused the Word spelling & grammar checker to complain that, for all the replaced opening quotes, the adjacent space should go after the quote character. This made me realize that the opening & closing quotes had different codes, though in my document they looked the same. Only when I enlarged the font, was a difference between them discernable.

Also, only the opening ?-in-a-rectangle was replaced by the closing ".  This made me very suspicious, so I used the first macro to display the code of the closing ?-in-a-rectangle - it also was different from the opening character code, which is what I'd used in the second macro.

So I undid all the replacements, and ran the second macro again, twice, each time with the corresponding codes for the opening & closing characters. (8220 --> 147, 8221 --> 148, all (normal text) font)

This time the result was perfect.

many thanks!.

If you can use another library, you can use apache commons https://commons.apache.org/proper/commons-text/javadocs/api-release/org/apache/commons/text/StringEscapeUtils.html

String dirtyString = "Colocaci\u00F3n";
String cleanString = StringEscapeUtils.unescapeJava(dirtyString);
//cleanString = "Colocación"

The package unidecode worked best for me:

from unidecode import unidecode
text = "Björn, Łukasz and Σωκράτης."
print(unidecode(text))
# ==> Bjorn, Lukasz and Sokrates.

You might need to install the package:

pip install unidecode

The above solution is easier and more robust than encoding (and decoding) the output of unicodedata.normalize(), as suggested by other answers.

# This doesn't work as expected:
ret = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore')
print(ret)
# ==> b'Bjorn, ukasz and .'
# Besides not supporting all characters, the returned value is a
# bytes object in python3. To yield a str type:
ret = ret.decode("utf8") # (not required in python2)