To answer the original question: here is how you decode utf-8 in javascript:
http://ecmanaut.blogspot.ca/2006/07/encoding-decoding-utf8-in-javascript.html
Specifically,
function encode_utf8(s) {
return unescape(encodeURIComponent(s));
}
function decode_utf8(s) {
return decodeURIComponent(escape(s));
}
We have been using this in our production code for 6 years, and it has worked flawlessly.
Note, however, that escape() and unescape() are deprecated. See this.
Answer from CpnCrunch on Stack Overflowunicode - Decode UTF-8 with Javascript - Stack Overflow
java - string decode utf-8 - Stack Overflow
Decoding UTF-8 strings in Python - Stack Overflow
Trouble Decoding from UTF-8
To answer the original question: here is how you decode utf-8 in javascript:
http://ecmanaut.blogspot.ca/2006/07/encoding-decoding-utf8-in-javascript.html
Specifically,
function encode_utf8(s) {
return unescape(encodeURIComponent(s));
}
function decode_utf8(s) {
return decodeURIComponent(escape(s));
}
We have been using this in our production code for 6 years, and it has worked flawlessly.
Note, however, that escape() and unescape() are deprecated. See this.
This should work:
// http://www.onicos.com/staff/iz/amuse/javascript/expert/utf.txt
/* utf.js - UTF-8 <=> UTF-16 convertion
*
* Copyright (C) 1999 Masanao Izumo <iz@onicos.co.jp>
* Version: 1.0
* LastModified: Dec 25 1999
* This library is free. You can redistribute it and/or modify it.
*/
function Utf8ArrayToStr(array) {
var out, i, len, c;
var char2, char3;
out = "";
len = array.length;
i = 0;
while(i < len) {
c = array[i++];
switch(c >> 4)
{
case 0: case 1: case 2: case 3: case 4: case 5: case 6: case 7:
// 0xxxxxxx
out += String.fromCharCode(c);
break;
case 12: case 13:
// 110x xxxx 10xx xxxx
char2 = array[i++];
out += String.fromCharCode(((c & 0x1F) << 6) | (char2 & 0x3F));
break;
case 14:
// 1110 xxxx 10xx xxxx 10xx xxxx
char2 = array[i++];
char3 = array[i++];
out += String.fromCharCode(((c & 0x0F) << 12) |
((char2 & 0x3F) << 6) |
((char3 & 0x3F) << 0));
break;
}
}
return out;
}
Check out the JSFiddle demo.
Also see the related questions: here and here
A string needs no encoding. It is simply a sequence of Unicode characters.
You need to encode when you want to turn a String into a sequence of bytes. The charset the you choose (UTF-8, cp1255, etc.) determines the Character->Byte mapping. Note that a character is not necessarily translated into a single byte. In most charsets, most Unicode characters are translated to at least two bytes.
Encoding of a String is carried out by:
String s1 = "some text";
byte[] bytes = s1.getBytes("UTF-8"); // Charset to encode into
You need to decode when you have а sequence of bytes and you want to turn them into a String. When yоu dо that you need to specify, again, the charset with which the bytеs were originally encoded (otherwise you'll end up with garblеd tеxt).
Decoding:
String s2 = new String(bytes, "UTF-8"); // Charset with which bytes were encoded
If you want to understand this better, a great text is "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)"
the core functions are getBytes(String charset) and new String(byte[] data). you can use these functions to do UTF-8 decoding.
UTF-8 decoding actually is a string to string conversion, the intermediate buffer is a byte array. since the target is an UTF-8 string, so the only parameter for new String() is the byte array, which calling is equal to new String(bytes, "UTF-8")
Then the key is the parameter for input encoded string to get internal byte array, which you should know beforehand. If you don't, guess the most possible one, "ISO-8859-1" is a good guess for English user.
The decoding sentence should be
String decoded = new String(encoded.getBytes("ISO-8859-1"));
It's an encoding error - so if it's a unicode string, this ought to fix it:
text.encode("windows-1252").decode("utf-8")
If it's a plain string, you'll need an extra step:
text.decode("utf-8").encode("windows-1252").decode("utf-8")
Both of these will give you a unicode string.
By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet:
>>> import chardet
>>> chardet.detect(u"And the Hip’s coming, too")
{'confidence': 0.5, 'encoding': 'windows-1252'}
You need to properly decode the source text. Most likely the source text is in UTF-8 format, not ASCII.
Because you do not provide any context or code for your question it is not possible to give a direct answer.
I suggest you study how unicode and character encoding is done in Python:
http://docs.python.org/2/howto/unicode.html
I have some code that ends up retrieving a bunch of strings, and each one is basically a utf-8 encoded symbol in string format, such as 'm\xc3\xbasica mexicana'. I want to encode this into bytes and then decode it as UTF-8 so that I can convert it into something like "música mexicana". I can achieve this if I start with a string that I create myself like below:
encoded_str = 'm\xc3\xbasica mexicana'
utf8_encoded = encoded_str.encode('raw_unicode_escape')
decoded_str = utf8_encoded.decode(encoding='UTF-8')
print(decoded_str)
# This prints "música mexicana", which is the desired resultBut in my actual code where I read the string from a source and don't create it myself the encoding always adds an extra backslash in front of the original string backslashes. Then when I decode it it just converts back to the original string without the second backslash.
# Exclude Artist pages
excluded_words = ['image', 'followers', 'googleapis']
excluded_words_found = any(word in hashtag for word in excluded_words)
if not excluded_words_found or len(hashtag) < 50:
# Encode string into bytes then utf decode it to convert characters with accents
hashtag = hashtag.encode('raw_unicode_escape')
hashtag = hashtag.decode(encoding='UTF-8')
# Add hashtag and uri to list
hashtags_uris.append((hashtag, uri))I've tried so many things, including using latin1 encoding instead of raw_unicode_escape and get the same result every time. Can anyone help me make sense of this?