You are indeed confused: bytes.decode - encodes the bytes as UTF-8 No, it decodes a byte sequence to a string, interpreting it as an UTF-8 encoded string. Strings are a sequence of code points (or characters) in Unicode. Unicode and UTF-8 are not the same thing! UTF-8 is the most popular, but still only one of many encoding schemes for Unicode. Take the string ABCÄĀ. This is represented by the unicode code point sequence U+0041 U+0042 U+0043 U+00c4 U+0100. As you see, unicode code points are more than 8 bits, so we need a way to store them in 8-bit byte sequences. One way to do that is to use UTF-8, which turns it into the byte sequence 0x41 0x42 0x43 0xc3 0x84 0xc4 0x80. As another example, UTF-16 would encode this as 0xff 0xfe 0x00 0x41 0x00 0x42 0x00 0x43 0x00 0xc4 0x01 0x00. Notice that except for the 0xff 0xfe BOM (Byte Order Mark) at the beginning, it's simply storing the 16 bit value of the code point in sequence (code points can have more than 16 bits, so it only works this way below 32768, similar to how UTF-8 doesn't change the representation below 128). The BOM is needed to know which of the two bytes represents the most significant bits, as in, is 0x12 0x34 representing 0x1234 or 0x3412. (Some machines store 16 bit values in the first way ("big endian"), others, like x86, in the other ("little endian")) There are also other encoding schemes, for example, if every unicode code point is below 256, you can use latin-1, which will just use the lower 8 bits of the unicode code point. To summarize: bytes.decode decodes a byte sequence into a string, which is an abstract representation of code points. str.encode turns this abstract representation into a byte sequence. Answer from Yoghurt42 on reddit.com
🌐
CodeShack
codeshack.io › home › tools › utf-8 decoder
UTF-8 Decoder - Convert UTF-8 Bytes (Hex, Binary, Decimal) to Text
Free UTF-8 decoder. Turn UTF-8 byte sequences in hexadecimal, binary or decimal back into readable text, with any separator or prefix. Fast, private and 100% in your browser.
🌐
Hixie
software.hixie.ch › utilities › cgi › unicode-decoder › utf8-decoder
utf8-decoder
A hex dump. Bytes are extracted two characters at a time, with all whitespace ignored, e.g.
Discussions

Why is encoding bytes to UTF-8 called decoding?
You are indeed confused: bytes.decode - encodes the bytes as UTF-8 No, it decodes a byte sequence to a string, interpreting it as an UTF-8 encoded string. Strings are a sequence of code points (or characters) in Unicode. Unicode and UTF-8 are not the same thing! UTF-8 is the most popular, but still only one of many encoding schemes for Unicode. Take the string ABCÄĀ. This is represented by the unicode code point sequence U+0041 U+0042 U+0043 U+00c4 U+0100. As you see, unicode code points are more than 8 bits, so we need a way to store them in 8-bit byte sequences. One way to do that is to use UTF-8, which turns it into the byte sequence 0x41 0x42 0x43 0xc3 0x84 0xc4 0x80. As another example, UTF-16 would encode this as 0xff 0xfe 0x00 0x41 0x00 0x42 0x00 0x43 0x00 0xc4 0x01 0x00. Notice that except for the 0xff 0xfe BOM (Byte Order Mark) at the beginning, it's simply storing the 16 bit value of the code point in sequence (code points can have more than 16 bits, so it only works this way below 32768, similar to how UTF-8 doesn't change the representation below 128). The BOM is needed to know which of the two bytes represents the most significant bits, as in, is 0x12 0x34 representing 0x1234 or 0x3412. (Some machines store 16 bit values in the first way ("big endian"), others, like x86, in the other ("little endian")) There are also other encoding schemes, for example, if every unicode code point is below 256, you can use latin-1, which will just use the lower 8 bits of the unicode code point. To summarize: bytes.decode decodes a byte sequence into a string, which is an abstract representation of code points. str.encode turns this abstract representation into a byte sequence. More on reddit.com
🌐 r/learnpython
7
21
March 30, 2022
python - Decode byte with UTF-8 - Stack Overflow
I am currently querying a kdb system and it is returning the data in bytes. Specifically in one column, I am getting a byte object that looks likes this b'US $ to UK \xa3 (TTF)' If I want to deco... More on stackoverflow.com
🌐 stackoverflow.com
python - Why can't I decode any byte using utf-8? - Stack Overflow
Not an expert in encoding, trying to learn. I got a file in latin encoding, when trying to read it and decode using 'utf-8' I get the following error: UnicodeDecodeError: 'utf-8' codec can't decode... More on stackoverflow.com
🌐 stackoverflow.com
python - Python3: Decode UTF-8 bytes converted as string - Stack Overflow
Suppose I have something like: a = "Gżegżółka" a = bytes(a, 'utf-8') a = str(a) which returns string in form: b'G\xc5\xbceg\xc5\xbc\xc3\xb3\xc5\x82ka' Now it's send as simple string (I get it as More on stackoverflow.com
🌐 stackoverflow.com
People also ask

Can it decode bytes with no separators?
Yes. Select None and it reads two hex digits, eight binary digits or three octal digits per byte. Decimal needs a separator because decimal bytes vary in length.
🌐
codeshack.io
codeshack.io › home › tools › utf-8 decoder
UTF-8 Decoder - Convert UTF-8 Bytes (Hex, Binary, Decimal) to Text
Why do I get an invalid UTF-8 message?
UTF-8 has strict rules about how multi-byte characters are built. If the bytes are in the wrong order, incomplete, or simply not valid UTF-8, the decoder flags it instead of inventing characters. Double-check the format and separator you selected.
🌐
codeshack.io
codeshack.io › home › tools › utf-8 decoder
UTF-8 Decoder - Convert UTF-8 Bytes (Hex, Binary, Decimal) to Text
What is the difference between this and an ASCII decoder?
An ASCII decoder treats each number as one character code. A UTF-8 decoder treats the numbers as bytes and combines multi-byte sequences into single characters, which matters for anything beyond basic English.
🌐
codeshack.io
codeshack.io › home › tools › utf-8 decoder
UTF-8 Decoder - Convert UTF-8 Bytes (Hex, Binary, Decimal) to Text
🌐
Rosetta Code
rosettacode.org › wiki › UTF-8_encode_and_decode
UTF-8 encode and decode - Rosetta Code
1 month ago - ; is_valid_utf8_continuation_byte c2; is_valid_utf8_continuation_byte c3; is_valid_utf8_continuation_byte c4; u == extended_utf8_char_5byte_decoding (c0, c1, c2, c3, c4)} EXTENDED_UTF8_CHAR_5byte (5, u, c0, c1, c2, c3, c4, ~1) // | {u, c0, c1, c2, c3, c4, c5 : int | 0 <= u; u <= 0x7FFFFFFF; is_extended_utf8_6byte_first_byte c0; is_valid_utf8_continuation_byte c1; is_valid_utf8_continuation_byte c2; is_valid_utf8_continuation_byte c3; is_valid_utf8_continuation_byte c4; is_valid_utf8_continuation_byte c5; u == extended_utf8_char_6byte_decoding (c0, c1, c2, c3, c4, c5)} EXTENDED_UTF8_CHAR_6byte
🌐
Boxentriq
boxentriq.com › home › encodings › utf-8 decoder
UTF-8 Decoder | Boxentriq
Each Unicode character is encoded using 1-4 bytes. Standard 7-bit ASCII characters are always encoded as a single byte in UTF-8, making the UTF-8 encoding backwards compatible with ASCII.
🌐
Reddit
reddit.com › r/learnpython › why is encoding bytes to utf-8 called decoding?
r/learnpython on Reddit: Why is encoding bytes to UTF-8 called decoding?
March 30, 2022 -

This always confuses me every single time. I remember it by doing the opposite of what I intuitively think it should be. Perhaps I'm the only one?

bytes.decode - encodes the bytes as UTF-8

str.encode - decodes "removes" the UTF-8 encoding

There is also the abbreviation BADTIE: Bytes Are Decoded, Text Is Encoded. To help to remember it.

Top answer
1 of 4
25
You are indeed confused: bytes.decode - encodes the bytes as UTF-8 No, it decodes a byte sequence to a string, interpreting it as an UTF-8 encoded string. Strings are a sequence of code points (or characters) in Unicode. Unicode and UTF-8 are not the same thing! UTF-8 is the most popular, but still only one of many encoding schemes for Unicode. Take the string ABCÄĀ. This is represented by the unicode code point sequence U+0041 U+0042 U+0043 U+00c4 U+0100. As you see, unicode code points are more than 8 bits, so we need a way to store them in 8-bit byte sequences. One way to do that is to use UTF-8, which turns it into the byte sequence 0x41 0x42 0x43 0xc3 0x84 0xc4 0x80. As another example, UTF-16 would encode this as 0xff 0xfe 0x00 0x41 0x00 0x42 0x00 0x43 0x00 0xc4 0x01 0x00. Notice that except for the 0xff 0xfe BOM (Byte Order Mark) at the beginning, it's simply storing the 16 bit value of the code point in sequence (code points can have more than 16 bits, so it only works this way below 32768, similar to how UTF-8 doesn't change the representation below 128). The BOM is needed to know which of the two bytes represents the most significant bits, as in, is 0x12 0x34 representing 0x1234 or 0x3412. (Some machines store 16 bit values in the first way ("big endian"), others, like x86, in the other ("little endian")) There are also other encoding schemes, for example, if every unicode code point is below 256, you can use latin-1, which will just use the lower 8 bits of the unicode code point. To summarize: bytes.decode decodes a byte sequence into a string, which is an abstract representation of code points. str.encode turns this abstract representation into a byte sequence.
2 of 4
11
Because in Python 3 the strings are in unicode by default, and bytes are an optional encoding, if you want to deal with things on that level.
🌐
PHP
php.net › manual › en › function.utf8-decode.php
PHP: utf8_decode - Manual
#[\Deprecated] function utf8_decode(string $string): string · This function converts the string string from the UTF-8 encoding to ISO-8859-1. Bytes in the string which are not valid UTF-8, and UTF-8 characters which do not exist in ISO-8859-1 ...
Find elsewhere
🌐
Wikipedia
en.wikipedia.org › wiki › UTF-8
UTF-8 - Wikipedia
3 days ago - In some cases the user will want to ensure no normalization is done; for this "utf8-c8" can be used. That UTF-8 Clean-8 variant, implemented by Raku, is an encoder/decoder that preserves bytes as is (even illegal UTF-8 sequences) and allows for Normal Form Grapheme synthetics.
🌐
Teleport
goteleport.com › home › resources › tools › utf-8 decoder | instantly transform utf-8 hex to readable text
UTF-8 Decoder | Instantly Transform UTF-8 Hex to Readable Text | Teleport
One approach is to use a buffering technique, where you maintain a buffer of bytes and only decode complete characters. Here's an example implementation in Python: def decode_utf8_stream(stream): buffer = b'' for chunk in stream: buffer += chunk try: decoded, consumed = buffer.decode('utf-8'), len(buffer) yield decoded buffer = b'' except UnicodeDecodeError: if len(buffer) > 4: raise continue if buffer: yield buffer.decode('utf-8', errors='replace')
🌐
Microsoft Learn
learn.microsoft.com › en-us › dotnet › api › system.text.utf8encoding.getdecoder
UTF8Encoding.GetDecoder Method (System.Text) | Microsoft Learn
The decoder converts a sequence of bytes into a sequence of characters. using System; using System.Text; class UTF8EncodingExample { public static void Main() { Char[] chars; Byte[] bytes = new Byte[] { 99, 204, 128, 234, 130, 160 }; Decoder ...
🌐
Online Tools
onlinetools.com › utf8 › convert-bytes-to-utf8
Convert Bytes to UTF8 – Online UTF8 Tools
World's simplest browser-based bytes to UTF8 string converter. Just import your raw bytes in the editor on the left and you will instantly get a UTF8 representation of these bytes on the right.
🌐
Browserling
browserling.com › tools › utf8-decode
UTF-8 Decode - Convert UTF-8 to Text - Online - Browserling Web Developer Tools
Also, you can use this program to verify the given UTF8 has been correctly encoded. UTF8 allows only specific byte sequences and if the bytes have errors, then you'll see them output here. Another use case is debugging UTF8-encoded data.
🌐
Qodex
qodex.ai › all-tools › utf8-decoder
UTF8 Decoder
UTF-8 decoding is the process of converting a sequence of hexadecimal bytes (encoded using UTF-8) back into human-readable text. UTF-8 (Unicode Transformation Format - 8 bit) is the most widely used character encoding format on the web.
🌐
Educative
educative.io › answers › what-is-bytesdecode-in-python
What is bytes.decode() in Python?
Line 3: We encode string, cast to byte object. Line 5: We use the decode() method with utf8 encoding scheme to transform from encoded values to a string object.
🌐
CodeShack
codeshack.io › home › tools › utf-8 encoder
UTF-8 Encoder - Convert Text to UTF-8 Bytes (Hex, Binary, Decimal)
Free UTF-8 encoder. Convert any text into its UTF-8 byte sequence in hexadecimal, binary or decimal, with custom separators, prefixes and hex case. Fast, private and 100% in your browser.
🌐
JSON
json.org › JSON_checker › utf8_decode.c
utf8_decode.c
IN NO EVENT SHALL THE AUTHORS OR ... OR OTHER DEALINGS IN THE SOFTWARE. */ #include "utf8_decode.h" /* Very Strict UTF-8 Decoder UTF-8 is a multibyte character encoding of Unicode. A character can be represented by 1-4 bytes....
Top answer
1 of 1
1

There are two different concepts being confused.

A character set is an ordering of characters, numbered from 0 to ...

An encoding is a way of representing the numbers in a character set as a sequence of bytes.


For a character set with at most 256 characters, a trivial single-byte encoding is possible.

For larger character sets, multi-byte encodings are required. They can be split into two types: fixed size, where every character uses the same number of bytes, and variable size, where different numbers of bytes are used to represent different characters.

Examples of fixed-size encodings are all single-byte encodings and UTF-32.

Examples of variable-sized encodings are UTF-8 and the various UTF-16 encodings.


Latin-1 is both a character set (containing the ASCII characters and its additional characters for writing Western European languages) and (having 256 characters) its corresponding single-byte encoding

Unicode is a character set containing (or aiming to contain) all characters to write all known languages. It is, not surprisingly, much larger than 256 characters.

UTF-8 is just one multibyte encoding of Unicode, and a variable sized one. The first byte of each UTF-8 sequence tells you how may additional bytes follow it to encode a single Unicode code point.

Unicode and Latin-1 (the character set) coincide for their first 256 code points. That is, Latin-1 is a subset of Unicode.

UTF-8 and Latin-1 coincide for their first 128 sequences. After that, they diverge. In UTF-8, code points 128 through 255 require two bytes. The first byte has the form 110xxxxx, the second 10xxxxxx. The 11 x bits are free for encoding the rest of the Latin-1 subset (plus additional blocks) of Unicode.

The byte 0xa0 is not valid UTF-8, as its binary expansion is 10100000. The first byte of a multibyte UTF-8 sequence always starts with at least two 1s (intuitively, the number of 1s indicates the number of bytes in the sequence).

🌐
Testsigma
testsigma.com › home › free tools › utf8 decode
UTF-8 Decoder Online | Free Decode & Validate Tool
It can represent every character in the Unicode standard using one to four bytes per character, making it the default for HTML5, JSON, XML, APIs, and virtually every modern web system.
🌐
DomSignal
domsignal.com › home › tools › utf-8 decoder & encoder
UTF-8 Decoder - Encode and Decode UTF-8 Text
Decode UTF-8 hex bytes to text or encode text to UTF-8 byte sequences online. Client-side converter with no API request required.
Top answer
1 of 2
7

If you want to encode and decode text, that's what the encode and decode methods are for:

>>> a = "Gżegżółka"
>>> b = a.encode('utf-8')
>>> b
b'G\xc5\xbceg\xc5\xbc\xc3\xb3\xc5\x82ka'
>>> c = b.decode('utf-8')
>>> c
'Gżegżółka'

Also, notice that UTF-8 is already the default, so you can just do this:

>>> b = a.encode()
>>> c = b.decode()

The only reason you need to specify arguments is:

  • You need to use some other encoding instead of UTF-8,
  • You need to specify a specific error handler, like 'surrogatereplace' instead of 'strict', or
  • Your code has to run in Python 3.0-3.1 (which almost nobody used).

However, if you really want to, you can do what you were already doing; you just need to explicitly specify the encoding in the str call, just as you did in the bytes call:

>>> a = "Gżegżółka"
>>> b = bytes(a, 'utf-8')
>>> b
b'G\xc5\xbceg\xc5\xbc\xc3\xb3\xc5\x82ka'
>>> c = str(b, 'utf-8')
>>> c

Calling str on a bytes object without an encoding, as you were doing, doesn't decode it, and doesn't raise an exception like calling bytes on a str without an encoding, because the main job of str is to give you a string representation of the object—and the best string representation of a bytes object is that b'…'.

2 of 2
0

I found it. The simplest way to convert string representation of bytes to bytes again is through the eval statement:

a = "Gżegżółka"
a = bytes(a, 'utf-8')
a = str(a) #this is the input we deal with

a = eval(a) #that's how we transform a into bytes
a = str(a, 'utf-8') #...and now we convert it into string

print(a)