1. Decode the string to Unicode. Assuming it's UTF-8-encoded:

   str.decode("utf-8")
   

2. Call the replace method and be sure to pass it a Unicode string as its first argument:

   str.decode("utf-8").replace(u"\u2022", "*")
   

3. Encode back to UTF-8, if needed:

   str.decode("utf-8").replace(u"\u2022", "*").encode("utf-8")
   

(Fortunately, Python 3 puts a stop to this mess. Step 3 should really only be performed just prior to I/O. Also, mind you that calling a string str shadows the built-in type str.)

You can try using Linq:

  using System.Linq;

  ...

  string source = "İntersport";

  // you may want to change 255 into 127 if you want standard ASCII table
  string target = string.Concat(source
    .Select(c => c < 32 || c > 255  
       ? "\\u" + ((int)c).ToString("x4") // special symbol: command one or above Ascii 
       : c.ToString()));                 // within ascii table [32..255]

  // \u0130ntersport
  Console.Write(target);

Edit: No Linq solution:

  string source = "İntersport";

  StringBuilder sb = new StringBuilder();

  foreach (char c in source) 
    if (c < 32 || c > 255)
      sb.Append("\\u" + ((int)c).ToString("x4"));
    else
      sb.Append(c);

  string target = sb.ToString();

You can iterate while there are " in the string and in every iteration replace one pair of quotes:

zitat = 'Laut Durkheim ist ein "soziologischer Tatbestand jede mehr oder weniger [...] unabhängiges Eigenleben besitzt"'

print(f"Before replace: {zitat}")

while "\"" in zitat:
    zitat = zitat.replace("\"", "\u201e", 1)
    zitat = zitat.replace("\"", "\u201c", 1)

print(f"After replace: {zitat}")

The 1 as third argument in replace() is important to replace only the first ocurrence of the ". This should give a correct output for any string with an even number of ".

Output:

Before replace: Laut Durkheim ist ein "soziologischer Tatbestand jede mehr oder weniger [...] unabhängiges Eigenleben besitzt"
After replace: Laut Durkheim ist ein „soziologischer Tatbestand jede mehr oder weniger [...] unabhängiges Eigenleben besitzt“

I believe the problem is that Vim not opening the file using the right encoding. Your file is in encoding cp1252 but Vim doesn't detect it and guess utf-8.

I would propose to open the file with the ++enc=cp1252 flag.

:e ++enc=cp1252 filepath

To improve the way Vim guess the encoding you can add the following lines (Vim would first try to open as cp1252 and then as utf-8):

:set fileencodings=cp1252,utf-8

You can solve this problem by dividing it into multiple steps:

• Convert the Unicode code points to actual entities. This can be easily achieved using preg_replace(). For an explanation of how the regex works, see my answer here.

• Now you will have a set of characters like ü. These are HTML entities. To convert them into their corresponding character forms, use html_entity_decode().

• You will now have a UTF-8 string. You need to convert it into ISO-8859-1 (Official ISO 8-bit Latin-1). The //TRANSLIT part is to enable transileration. If this is enabled, when a character can't be represented in the target charset, it will try to approximate the result.

Code:

// Set the locale to something that's UTF-8 capable
setlocale(LC_ALL, 'en_US.UTF-8');

$str = "u\u0308 a\u0302 a\u0308 o\u0300";

// Convert the codepoints to entities
$str = preg_replace("/\\\\u([0-9a-fA-F]{4})/", "&#x\\1;", $str);

// Convert the entities to a UTF-8 string
$str = html_entity_decode($str, ENT_QUOTES, 'UTF-8');

// Convert the UTF-8 string to an ISO-8859-1 string
echo iconv("UTF-8", "ISO-8859-1//TRANSLIT", $str);

Output:

u a a o

Demo

I wonder how do I search and replace unicode character xE5" with æ

Note that æ is actually Unicode 00E6 not 00E5.

Search and replace is not the right way to get the correct characters displayed.

<?xml version="1.0" encoding="utf-8"?>

The above states the encoding is utf-8 but the file is actually encoded as ANSI.

You need to convert the file correctly to UTF-8, as follows:

1. Open Testfile.xlf

2. File looks like:

   

   Unicode is incorrectly displayed.

3. Menu > Encoding > Select Encode in ANSI
   
   

   

4. File looks like:

   

   Unicode is correctly displayed.

5. Select all file contents (ctrl+a)

6. Menu > Encoding > Select Convert to UTF-8
   
   

   

7. Save the File (ctrl+s)

8. Close and reopen.

9. File is now correctly encoded as UTF-8 and Unicode characters display correctly.

How can you see the file is actually ANSI?

The cygwin file utility shows this (before and after conversion):

DavidPostill@Hal /f/test
$ file -i Testfile*.xlf
Testfile.xlf:          application/xml; charset=iso-8859-1
TestfileConverted.xlf: application/xml; charset=utf-8

I use a javascript bookmarklet for typing unicode symbols at math.stackexchange.com. Mathjax renders most unicode the same as the corresponding latex macros. For example $ℝ$ and $\mathbb{R}$ give the same result. I like the way tex code stays more compact and readable with unicode symbols.

I think this code is able to do what you want. I like to use not too many keystrokes, so instead of \alpha I use \a to produce α. You can modify this script to your own needs, and then convert it to a bookmarklet, using this website for example: http://jasonmillerdesign.com/Free_Stuff/Instant_Bookmarklet_Converter

If you want to use this script on a website without jquery, then you first need to run this bookmarklet: http://www.learningjquery.com/2006/12/jquerify-bookmarklet/

jQuery.fn.autocorrect = function(options)
{
    if ("text" != jQuery(this).attr("type") && !jQuery(this).is("textarea"))
    {
        return;
    }
var defaults = {
        corrections: {
            a: "α",
            b: "β",
            c: "γ",
            d: "δ",
            e: "ϵ",
            emp : "∅",
            f: "\\frac{}{}",
            in : "∈",
            s: "σ",
            t: "\\text{}",
            tau : "τ",
            th : "θ",
            p: "π",
            pm : "±",
            o : "ω",
            O : "Ω",
            r : "ρ",
            A : "∀",
            E : "∃",
            R: "ℝ",
            C: "ℂ",
            H: "ℍ",
            N: "ℕ",
            Q: "ℚ",
            Z: "ℤ",
            int: "\\int_{}^{}",
            inf : "∞",
            sum : "\\sum_{}^{}",
            "-1": "^{-1}",
            ph: "ϕ",
            ch: "χ",
            ps: "ψ",
            leq : "≥",
            xi : "ξ", 
            geq : "≤",
            "/=" : "≠",
            "==" : "≡",
            "<" : "\\langle {} \\rangle",
            "->" : "→",
            "=>" : "⇒",
            "<=" : "⇐",
            "<>" : "⇔",
            "sq" : "\\sqrt{}"
    }
};
if (options && options.corrections)
{
    options.corrections = jQuery.extend(defaults.corrections, options.corrections);
}
var opts = jQuery.extend(defaults, options);
getCaretPosition = function(oField)
{
    var iCaretPos = 0;
    if (document.selection)
    {
        var oSel = document.selection.createRange();
        oSel.moveStart("character", 0 - oField.value.length);
        iCaretPos = oSel.text.length;
    }
    else if (oField.selectionStart || oField.selectionStart == "0")
    {
        iCaretPos = oField.selectionStart;
    }
    return (iCaretPos);
}
function setCaretPosition (oField, iCaretPos)
{
    if (document.selection)
    {
        var oSel = document.selection.createRange();
        oSel.moveStart("character", 0 - oField.value.length);
        oSel.moveStart("character", iCaretPos);
        oSel.moveEnd("character", 0);
    }
    else if (oField.selectionStart || oField.selectionStart == "0")
    {
        oField.selectionStart = iCaretPos;
        oField.selectionEnd = iCaretPos;
    }
}
this.keyup(function(e)
{
    if (32 != e.keyCode)
    {
        return;
    }
    var caretPosition = (getCaretPosition(this) - 1);
    if (1 > caretPosition)
    {
        return;
    }
    var valueOfField = this.value;
    var stringUptoCaretPosition = (valueOfField).substr(0, caretPosition);
    if (" " == stringUptoCaretPosition.charAt(caretPosition - 1))
    {
        return;
    }
    var beginIndex = stringUptoCaretPosition.lastIndexOf('\\');
    if (beginIndex < stringUptoCaretPosition.lastIndexOf(' '))
    {
        return;
    }
    var stringToSearch = stringUptoCaretPosition.substring(beginIndex+1);
    var stringNotToSearch = stringUptoCaretPosition.substring(0, beginIndex);
    if (!opts.corrections[stringToSearch])
    {
        return;
    }
    var stringToReplace = opts.corrections[stringToSearch];
    stringUptoCaretPosition = stringNotToSearch+ stringToReplace;
    var stringFromCaretPositionUptoEnd = (valueOfField).substr(caretPosition+1);
    this.value = (stringUptoCaretPosition + stringFromCaretPositionUptoEnd);
    if (stringToReplace.indexOf("{}")!=-1 )
    {
    setCaretPosition(this, stringUptoCaretPosition.indexOf("{}")+1);
    }
    else { setCaretPosition(this, stringUptoCaretPosition.length);}

});
};
$(document).ready(function()
{
    $("textarea").autocorrect();
});

You need to escape the backslash so that \u isn't treated as the beginning of a Python escape sequence.

"Czesć".replace("ć", "\\u0107")

Word plays internal tricks with Unicode symbols, and doesn't report the true character values. To make successful replacements, use the macros in the article https://wordmvp.com/FAQs/MacrosVBA/FindReplaceSymbols.htm.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import unicodedata
text = u'Cześć'
print unicodedata.normalize('NFD', text).encode('ascii', 'ignore')

That depends on what you mean. If you just want to get rid of them, do this:
(Update: Apparently you want to keep digits as well, use the second lines in that case)

String alphaOnly = input.replaceAll("[^a-zA-Z]+","");
String alphaAndDigits = input.replaceAll("[^a-zA-Z0-9]+","");

or the equivalent:

String alphaOnly = input.replaceAll("[^\\p{Alpha}]+","");
String alphaAndDigits = input.replaceAll("[^\\p{Alpha}\\p{Digit}]+","");

(All of these can be significantly improved by precompiling the regex pattern and storing it in a constant)

Or, with Guava:

private static final CharMatcher ALNUM =
  CharMatcher.inRange('a', 'z').or(CharMatcher.inRange('A', 'Z'))
  .or(CharMatcher.inRange('0', '9')).precomputed();
// ...
String alphaAndDigits = ALNUM.retainFrom(input);

But if you want to turn accented characters into something sensible that's still ascii, look at these questions:

• Converting Java String to ASCII
• Java change áéőűú to aeouu
• ń ǹ ň ñ ṅ ņ ṇ ṋ ṉ ̈ ɲ ƞ ᶇ ɳ ȵ --> n or Remove diacritical marks from unicode chars