It's an encoding error - so if it's a unicode string, this ought to fix it:
text.encode("windows-1252").decode("utf-8")
If it's a plain string, you'll need an extra step:
text.decode("utf-8").encode("windows-1252").decode("utf-8")
Both of these will give you a unicode string.
By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet:
>>> import chardet
>>> chardet.detect(u"And the Hip’s coming, too")
{'confidence': 0.5, 'encoding': 'windows-1252'}
It's an encoding error - so if it's a unicode string, this ought to fix it:
text.encode("windows-1252").decode("utf-8")
If it's a plain string, you'll need an extra step:
text.decode("utf-8").encode("windows-1252").decode("utf-8")
Both of these will give you a unicode string.
By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet:
>>> import chardet
>>> chardet.detect(u"And the Hip’s coming, too")
{'confidence': 0.5, 'encoding': 'windows-1252'}
You need to properly decode the source text. Most likely the source text is in UTF-8 format, not ASCII.
Because you do not provide any context or code for your question it is not possible to give a direct answer.
I suggest you study how unicode and character encoding is done in Python:
http://docs.python.org/2/howto/unicode.html
UTF-8 decoding
how to decode UTF-8?
The use of open(encoding="utf-8")
Hi! So I'm trying to decode some utf-8 strings in Python. However some bring up an error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb3 in position 3: invalid start byte or something along the lines of that. Here's the function: def b_numstring(f):
name_len = int.from_bytes(f.read(4), byteorder='big')
string = f.read(name_len).decode('utf-8')
return string Any help would be appreciated. Thanks!
Hello,
I've been struggling to figure this one out. This is pulling from the google API and extracting from my gmail account the message. As per the documentation, I have been able to locate the body of the message as per some of their references here.
Below is the snippet of how it looks. I've shortened it to be more user friendly.
'headers': [{'name': 'Content-Type', 'value': 'text/plain; charset="utf-8"'},
{'name': 'Content-Transfer-Encoding', 'value': '8bit'}],
'body': {'size': 3381,
'data': 'Shortened gobbledigookDQcz1lNj=='}}]}, 'sizeEstimate': 22874}
import base64 body=email_message['payload']['parts'][0]['body']['data'] translated_body = base64.b64decode(bytes(body), 'UTF-8')
From what I can see, my body variable shoots out the long alphanumeric string that should be the email body. However, I keep getting this error:
TypeError: string argument without an encoding
Any idea why? If it helps to include more code I can, but I think this is the only relevant part that is giving me issues.
Figured it out through more searching on the web!
msg_str = str(base64.urlsafe_b64decode(message['raw'].encode('ASCII')),'UTF-8')
mime_msg = email.message_from_string(msg_str)When working with files, I sometimes run into UnicodeDecodeError. Apparently this is because not all unicode characters can be converted into strings, and I saw that some people solved this by adding encoding="utf-8" as a parameter, but have anyone got an idea on what this does?