It's an encoding error - so if it's a unicode string, this ought to fix it:
text.encode("windows-1252").decode("utf-8")
If it's a plain string, you'll need an extra step:
text.decode("utf-8").encode("windows-1252").decode("utf-8")
Both of these will give you a unicode string.
By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet:
>>> import chardet
>>> chardet.detect(u"And the Hip’s coming, too")
{'confidence': 0.5, 'encoding': 'windows-1252'}
It's an encoding error - so if it's a unicode string, this ought to fix it:
text.encode("windows-1252").decode("utf-8")
If it's a plain string, you'll need an extra step:
text.decode("utf-8").encode("windows-1252").decode("utf-8")
Both of these will give you a unicode string.
By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet:
>>> import chardet
>>> chardet.detect(u"And the Hip’s coming, too")
{'confidence': 0.5, 'encoding': 'windows-1252'}
You need to properly decode the source text. Most likely the source text is in UTF-8 format, not ASCII.
Because you do not provide any context or code for your question it is not possible to give a direct answer.
I suggest you study how unicode and character encoding is done in Python:
http://docs.python.org/2/howto/unicode.html
python - Python3: Decode UTF-8 bytes converted as string - Stack Overflow
utf 8 - Decoding utf-8 String using python 3.6 - Stack Overflow
UTF-8 decoding
Help with utf-8 encode / decode
If you want to encode and decode text, that's what the encode and decode methods are for:
>>> a = "Gżegżółka"
>>> b = a.encode('utf-8')
>>> b
b'G\xc5\xbceg\xc5\xbc\xc3\xb3\xc5\x82ka'
>>> c = b.decode('utf-8')
>>> c
'Gżegżółka'
Also, notice that UTF-8 is already the default, so you can just do this:
>>> b = a.encode()
>>> c = b.decode()
The only reason you need to specify arguments is:
- You need to use some other encoding instead of UTF-8,
- You need to specify a specific error handler, like
'surrogatereplace'instead of'strict', or - Your code has to run in Python 3.0-3.1 (which almost nobody used).
However, if you really want to, you can do what you were already doing; you just need to explicitly specify the encoding in the str call, just as you did in the bytes call:
>>> a = "Gżegżółka"
>>> b = bytes(a, 'utf-8')
>>> b
b'G\xc5\xbceg\xc5\xbc\xc3\xb3\xc5\x82ka'
>>> c = str(b, 'utf-8')
>>> c
Calling str on a bytes object without an encoding, as you were doing, doesn't decode it, and doesn't raise an exception like calling bytes on a str without an encoding, because the main job of str is to give you a string representation of the object—and the best string representation of a bytes object is that b'…'.
I found it. The simplest way to convert string representation of bytes to bytes again is through the eval statement:
a = "Gżegżółka"
a = bytes(a, 'utf-8')
a = str(a) #this is the input we deal with
a = eval(a) #that's how we transform a into bytes
a = str(a, 'utf-8') #...and now we convert it into string
print(a)
Hi! So I'm trying to decode some utf-8 strings in Python. However some bring up an error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb3 in position 3: invalid start byte or something along the lines of that. Here's the function: def b_numstring(f):
name_len = int.from_bytes(f.read(4), byteorder='big')
string = f.read(name_len).decode('utf-8')
return string Any help would be appreciated. Thanks!
Fun project but HELP!
I'm trying to build a small tool that logs into a Gmail account and pulls out all the emails from any specified folder (label) and places relevant data into a data frame for analysis.
All this is working splendidly except for emails that have subjects that are non-'utf-8' (e.g. french accents, etc.). This sucks as I have a lot of french in the email subjects and it kills my code.
I've been through endless documentation on encoding, decoding, unicode, etc. but nothing I do seems to work.
For example the subject:
"Après un été marqué par une chute de popularité"
gets interpreted as:
=?UTF-8?B?QXByw6hzIHVuIMOpdMOpIG1hcnF1w6kgcGFyIHVuZSBjaHV0ZSBkZSBwb3B1bGFyaXTDqQ==?=
Ay subject without extended characters renders just fine. I've been tearing my hair out for 24 hours on this one, encoding, decoding, throwing modules at the problem. I can't believe it's actually this challenging. This must be the most normal thing! What am I missing?
Subject = "Après un été marqué par une chute de popularité"
Here's the main issue:
raw_email_str = data[0][1].decode('utf-8')
email_message = email.message_from_string(raw_email_str)
msg_subj = email_message['Subject']
print (msg_subj)Output:
=?UTF-8?B?QXByw6hzIHVuIMOpdMOpIG1hcnF1w6kgcGFyIHVuZSBjaHV0ZSBkZSBwb3B1bGFyaXTDqQ==?=
I have some code that ends up retrieving a bunch of strings, and each one is basically a utf-8 encoded symbol in string format, such as 'm\xc3\xbasica mexicana'. I want to encode this into bytes and then decode it as UTF-8 so that I can convert it into something like "música mexicana". I can achieve this if I start with a string that I create myself like below:
encoded_str = 'm\xc3\xbasica mexicana'
utf8_encoded = encoded_str.encode('raw_unicode_escape')
decoded_str = utf8_encoded.decode(encoding='UTF-8')
print(decoded_str)
# This prints "música mexicana", which is the desired resultBut in my actual code where I read the string from a source and don't create it myself the encoding always adds an extra backslash in front of the original string backslashes. Then when I decode it it just converts back to the original string without the second backslash.
# Exclude Artist pages
excluded_words = ['image', 'followers', 'googleapis']
excluded_words_found = any(word in hashtag for word in excluded_words)
if not excluded_words_found or len(hashtag) < 50:
# Encode string into bytes then utf decode it to convert characters with accents
hashtag = hashtag.encode('raw_unicode_escape')
hashtag = hashtag.decode(encoding='UTF-8')
# Add hashtag and uri to list
hashtags_uris.append((hashtag, uri))I've tried so many things, including using latin1 encoding instead of raw_unicode_escape and get the same result every time. Can anyone help me make sense of this?