It's an encoding error - so if it's a unicode string, this ought to fix it:
text.encode("windows-1252").decode("utf-8")
If it's a plain string, you'll need an extra step:
text.decode("utf-8").encode("windows-1252").decode("utf-8")
Both of these will give you a unicode string.
By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet:
>>> import chardet
>>> chardet.detect(u"And the Hip’s coming, too")
{'confidence': 0.5, 'encoding': 'windows-1252'}
Top answer 1 of 2
67
It's an encoding error - so if it's a unicode string, this ought to fix it:
text.encode("windows-1252").decode("utf-8")
If it's a plain string, you'll need an extra step:
text.decode("utf-8").encode("windows-1252").decode("utf-8")
Both of these will give you a unicode string.
By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet:
>>> import chardet
>>> chardet.detect(u"And the Hip’s coming, too")
{'confidence': 0.5, 'encoding': 'windows-1252'}
2 of 2
15
You need to properly decode the source text. Most likely the source text is in UTF-8 format, not ASCII.
Because you do not provide any context or code for your question it is not possible to give a direct answer.
I suggest you study how unicode and character encoding is done in Python:
http://docs.python.org/2/howto/unicode.html
GeeksforGeeks
geeksforgeeks.org › python › python-strings-decode-method
Python Strings decode() method - GeeksforGeeks
May 11, 2026 - Encoding converts a string into bytes and decode() brings it back to its original form. ... t = "Hello, Python!" e_t = t.encode('utf-8') print("Encoded:", e_t) d_t = e_t.decode('utf-8') print("Decoded:", d_t)
UTF-8 decoding
The error is telling you the bytes you read with the f.read(name_len) was not a utf-8 encoded string. Probably you should check what the contents of the file f is, and possibly print out name_len and f.read(name_len) to check what you read is what you thought it should be. More on reddit.com
7
3
June 24, 2024how to decode UTF-8?
I don’t have an interpreter with me right now on mobile but I could’ve sworn you could do: X=“test” X.encode(“utf-8”) X.decode(“utf-8”) More on reddit.com
7
6
June 20, 2018The use of open(encoding="utf-8")
open defaults to whatever encoding your system uses by default, so it can be anything from ASCII to ISO-8859-1. You can, however, make it use a specific encoding instead. utf-8 is useful as it has most characters. More on reddit.com
16
3
November 2, 202010:12
Easy Python tutorials | Encode and decode with UTF-8 in Python ...
03:22
utf 8 python decode - YouTube
10:18
Manually decoding UTF-8 - YouTube
05:46
Unicode (UTF-8) reading and writing to files in Python - YouTube
04:01
Understanding Python String Conversion Between utf-8 and ...
Decoding Binary Strings to UTF-8 in Python
Python documentation
docs.python.org › 3 › howto › unicode.html
Unicode HOWTO — Python 3.14.7 documentation
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte >>> b'\x80abc'.decode("utf-8", "replace") '\ufffdabc' >>> b'\x80abc'.decode("utf-8", "backslashreplace") '\\x80abc' >>> b'\x80abc'.decode("utf-8", "ignore") 'abc'
Mimo
mimo.org › glossary › python › string-decode
Python string decode(): Syntax, Usage, and Examples
Start your coding journey with Python. Learn basics, data types, control flow, and more ... message = b'Hello, world!' decoded_message = message.decode('utf-8') print(decoded_message) # Output: Hello, world!
DigitalOcean
digitalocean.com › community › tutorials › python-string-encode-decode
Python String Encode and Decode: Complete Guide | DigitalOcean
Master Python string encode() and decode() methods. Learn UTF-8, ASCII, Unicode handling, error handling modes, and practical encoding/decoding examples.
Python
docs.python.org › 3 › library › codecs.html
codecs — Codec registry and base classes
Creates a StreamRecoder instance which implements a two-way conversion: encode and decode work on the frontend — the data visible to code calling read() and write(), while Reader and Writer work on the backend — the data in stream. You can use these objects to do transparent transcodings, e.g., from Latin-1 to UTF-8 and back.
Reddit
reddit.com › r/learnpython › utf-8 decoding
r/learnpython on Reddit: UTF-8 decoding
June 24, 2024 -
Hi! So I'm trying to decode some utf-8 strings in Python. However some bring up an error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb3 in position 3: invalid start byte or something along the lines of that. Here's the function: def b_numstring(f):
name_len = int.from_bytes(f.read(4), byteorder='big')
string = f.read(name_len).decode('utf-8')
return string Any help would be appreciated. Thanks!
Top answer 1 of 5
3
The error is telling you the bytes you read with the f.read(name_len) was not a utf-8 encoded string. Probably you should check what the contents of the file f is, and possibly print out name_len and f.read(name_len) to check what you read is what you thought it should be.
2 of 5
1
the code itself looks fine if the file had the format you assume it does. I guess at doesn't look like that. Maybe just have a look with a hexeditor.
Medium
medium.com › the-dark-grimoire › decoding-utf-8-hex-strings-in-python-5a22b06d7018
Decoding UTF-8 Hex Strings in Python | by Yen Wang | The Dark Grimoire | Medium
May 17, 2026 - .decode("utf-8"). UTF-8 is a variable-width encoding. Single ASCII characters occupy one byte; characters in the range U+0080 to U+07FF use two bytes; U+0800 to U+FFFF (which covers the Braille Patterns block, U+2800 to U+28FF) use three bytes.
Honeybadger
honeybadger.io › blog › python-character-encoding
Python developer's guide to character encoding - Honeybadger Developer Blog
March 6, 2023 - To decode bytes into strings, call the decode() method and specify the type of character encoding you wish to use. Of course, we are using UTF-8.
Reddit
reddit.com › r/learnpython › how to decode utf-8?
r/learnpython on Reddit: how to decode UTF-8?
June 20, 2018 -
Hello,
I've been struggling to figure this one out. This is pulling from the google API and extracting from my gmail account the message. As per the documentation, I have been able to locate the body of the message as per some of their references here.
Below is the snippet of how it looks. I've shortened it to be more user friendly.
'headers': [{'name': 'Content-Type', 'value': 'text/plain; charset="utf-8"'},
{'name': 'Content-Transfer-Encoding', 'value': '8bit'}],
'body': {'size': 3381,
'data': 'Shortened gobbledigookDQcz1lNj=='}}]}, 'sizeEstimate': 22874}
import base64 body=email_message['payload']['parts'][0]['body']['data'] translated_body = base64.b64decode(bytes(body), 'UTF-8')
From what I can see, my body variable shoots out the long alphanumeric string that should be the email body. However, I keep getting this error:
TypeError: string argument without an encoding
Any idea why? If it helps to include more code I can, but I think this is the only relevant part that is giving me issues.
Figured it out through more searching on the web!
msg_str = str(base64.urlsafe_b64decode(message['raw'].encode('ASCII')),'UTF-8')
mime_msg = email.message_from_string(msg_str) Top answer 1 of 3
4
I don’t have an interpreter with me right now on mobile but I could’ve sworn you could do: X=“test” X.encode(“utf-8”) X.decode(“utf-8”)
2 of 3
2
From looking at base64.b64decode() i'd say, try this: body=email_message['payload']['parts'][0]['body']['data'] result = base64.b64decode(body) print(result) Though, maybe you might have to try one of the following: result = str(base64.b64decode(body)) or result = base64.b64decode(body).decode('utf-8') If you don't succeed, show me what it prints in the above cases or what errors you get. Also, try using bytes(body) like you did in your original example, as it might need to be given the result in bytes as suggested. (but i think it already is)
MojoAuth
mojoauth.com › character-encoding-decoding › utf-8-encoding--python
UTF-8 Encoding : Python | Encoding Solutions Across Programming Languages
\xe4\xbd\xa0\xe5\xa5\xbd\xef\xbc\x8c\xe4\xb8\x96\xe7\x95\x8c\xef\xbc\x81' # Decoding the bytes back to a string decoded_text = encoded_text.decode('utf-8') # Displaying the decoded string print(decoded_text) # Output: Hello, World! 你好，世界！ · In this example, the previously encoded bytes are decoded back to their original string format, demonstrating the seamless transition between encoding and decoding in Python.
GitHub
github.com › openai › openai-python
GitHub - openai/openai-python: The official Python library for the OpenAI API · GitHub
import base64 from openai import OpenAI client = OpenAI() prompt = "What is in this image?" with open("path/to/image.png", "rb") as image_file: b64_image = base64.b64encode(image_file.read()).decode("utf-8") response = client.responses.create( model="gpt-5.5", input=[ { "role": "user", "content": [ {"type": "input_text", "text": prompt}, {"type": "input_image", "image_url": f"data:image/png;base64,{b64_image}"}, ], } ], )
Author: openai
Python documentation
docs.python.org › 3 › library › venv.html
venv — Creation of virtual environments
""" progress = self.progress while True: s = stream.readline() if not s: break if progress is not None: progress(s, context) else: if not self.verbose: sys.stderr.write('.') else: sys.stderr.write(s.decode('utf-8')) sys.stderr.flush() stream.close() def install_script(self, context, name, url): _, _, path, _, _ = urlsplit(url) fn = os.path.split(path)[-1] binpath = context.bin_path distpath = os.path.join(binpath, fn) # Download script into the virtual environment's binaries folder urlretrieve(url, distpath) progress = self.progress if self.verbose: term = '\n' else: term = '' if progress is n
Medium
medium.com › @agustinb › introduction-to-unicode-and-utf-8-in-python-9e7a844edddd
Introduction to Unicode and UTF-8 in Python 2 | by agustinb | Medium
December 1, 2018 - Python 2 does implicit decoding in order to make a single Unicode object, but its default codec is ASCII (you can run sys.getdefaultencoding() to check). So, in the first example, ‘world’ wasn’t a problem but ‘π’ cannot be decoded using ASCII. Codec is a shortcut for Encoder/Decoder. >>> content = '\xcf\x80-zza'.decode('utf-8') # π-zza>>> type(content) <type 'unicode'>>>> print content π-zza>>> output_string = content.encode('utf-8')>>> type(output_string) <type 'str'>>>> output_string '\xcf\x80-zza' # bytes!
LabEx
labex.io › tutorials › python-how-to-use-python-utf8-encoding-451217
How to use Python UTF8 encoding | LabEx
UTF-8 (Unicode Transformation Format - 8-bit) is a widely used character encoding standard that supports virtually all characters and symbols from different languages worldwide. It is a variable-width character encoding capable of representing every character in the Unicode standard. ... Python 3 natively supports UTF-8 encoding, making it easy to work with international text.
Programiz
programiz.com › python-programming › methods › string › encode
Python String encode()
Online Python Online JavaScript Online SQL Online Java Online HTML Online C Online C++ Online C# Online PHP Online Swift Online Kotlin Online TypeScript Online Go Online Rust Online Scala Online Dart Online R Online Ruby ... The encode() method returns an encoded version of the given string. ... By default, the encode() method doesn't require any parameters. It returns an utf-8 encoded version of the string.
Medium
lynn-kwong.medium.com › understand-the-encoding-decoding-of-python-strings-unicode-utf-8-f6f97a909ee0
Understand the encoding/decoding of Python strings (Unicode/UTF-8) | by Lynn G. Kwong | Medium
August 25, 2023 - Actually, this is a major problem if you upgrade a Python 2 codebase to Python3. You would need to manually fix bytes/Unicode string-related problems, especially when you have encoding and decoding in your code.
Evanjones
evanjones.ca › python-utf8.html
How to Use UTF-8 with Python (evanjones.ca)
The next line stores the UTF-8 representation of u in the byte string backToBytes. Thankfully, everything in Python is supposed to treat Unicode strings identically to byte strings. However, you need to be careful in your own code when testing to see if an object is a string.
Reddit
reddit.com › r/learnpython › the use of open(encoding="utf-8")
r/learnpython on Reddit: The use of open(encoding="utf-8")
November 2, 2020 -
When working with files, I sometimes run into UnicodeDecodeError. Apparently this is because not all unicode characters can be converted into strings, and I saw that some people solved this by adding encoding="utf-8" as a parameter, but have anyone got an idea on what this does?
Top answer 1 of 5
2
open defaults to whatever encoding your system uses by default, so it can be anything from ASCII to ISO-8859-1. You can, however, make it use a specific encoding instead. utf-8 is useful as it has most characters.
2 of 5
1
there are different ways to represent characters as bytes. utf-8 is quite common, but there's also utf-16, latin1 etc. the correct solution will depend upon the file you're opening
SSOJet
ssojet.com › character-encoding-decoding › utf-8-in-python
UTF-8 in Python | Encoding Standards for Programming Languages
Use the .encode() method on a string to transform it into a bytes object, specifying the desired encoding. Conversely, the .decode() method on a bytes object converts it back into a Unicode string, again requiring the correct encoding.