🌐
Python documentation
docs.python.org › 3 › howto › unicode.html
Unicode HOWTO — Python 3.14.7 documentation
UTF-8 is one of the most commonly used encodings, and Python often defaults to using it. UTF stands for “Unicode Transformation Format”, and the ‘8’ means that 8-bit values are used in the encoding.
Discussions

unicode - python encoding utf-8 - Stack Overflow
But I dont understand why, but I got problem with encoding. My MySQL database is in utf8, or seems to be SQL query SHOW variables LIKE 'char%' returns me only utf8 or binary. ... Copy#!/usr/bin/python # -*- coding: utf-8 -*- def saveIndex(index,date): import MySQLdb as mdb import codecs sql ... More on stackoverflow.com
🌐 stackoverflow.com
Unicode (UTF-8) reading and writing to files in Python - Stack Overflow
The answer to your "So, what's the point…" question is "Mu." (since Python can read files encoded in UTF-8). For your second question: \xc3 is not part of the ASCII set. Perhaps you mean "8-bit encoding" instead. You are confused about Unicode and encodings; it's ok, many are. More on stackoverflow.com
🌐 stackoverflow.com
Working with UTF-8 encoding in Python source - Stack Overflow
You need not use unicode(), simply write string in UTF-8 encoding. 2011-06-09T08:03:22.567Z+00:00 ... In Python versions older than 3, you also need to prefix unicode string literals with "u": some_string = u'idzie wąż wąską dróżką'. 2011-06-09T08:06:28.08Z+00:00 ... on a diffrent string I am getting """UnicodeEncodeError: 'charmap' codec can't encode characters in position 1845-1846: character maps to """... does that mean ... More on stackoverflow.com
🌐 stackoverflow.com
Writing files with UTF-8 encoding?
First, are you using Python 2 or 3? How Python 3 handles string encodings is one of the primary differences between the two. Basically, in your program you manipulate unicode strings. In Python 2 they will be unicode objects and the literals will have the u prefix. In python 3 they will be str objects. Before you write them to a file, the string needs to be encoded into a byte-representation. You do this with just mystring.encode("UTF-8"). The result of that will be a byte string, bytes in Python 3 or str in Python 2. If you try and write a unicode string to a file directly, it will probably use some system default encoding, or error depending on how you're opening the file and your python version. Also relevant: are you using some XML library to write the file? How are you getting the chinese characters into your program? Are you sure they're not gibberish before you even write them out? I don't have enough information to answer those questions for you. More on reddit.com
🌐 r/learnpython
7
6
May 13, 2013
🌐
MojoAuth
mojoauth.com › character-encoding-decoding › utf-8-encoding--python
UTF-8 Encoding : Python | Encoding Solutions Across Programming Languages
Dive in and enhance your Python skills with our expert tips and practical examples! UTF-8 (8-bit Unicode Transformation Format) is a variable-width character encoding that can represent every character in the Unicode character set.
🌐
Honeybadger
honeybadger.io › blog › python-character-encoding
Python developer's guide to character encoding - Honeybadger Developer Blog
March 6, 2023 - UTF-8 is a standard and efficient encoding of Unicode strings that represents characters in one-, two-, three-, or four-byte units. Python uses UTF-8 by default, which means it does not need to be specified in every Python file.
🌐
GeeksforGeeks
geeksforgeeks.org › python › convert-a-string-to-utf-8-in-python
Convert a String to Utf-8 in Python - GeeksforGeeks
July 23, 2025 - Unicode Transformation Format 8 (UTF-8) is a widely used character encoding that represents each character in a string using variable-length byte sequences. In Python, converting a string to UTF-8 is a common task, and there are several simple ...
variable-width encoding (into one to four bytes) and transformation format of code points for the universal character set defined by ISO/IEC 10646 and The Unicode® Standard, compatible with ASCII
standard compressed utf 8
standard compressed utf 8
UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode Transformation Format – 8-bit. As of 2026, almost every webpage (99.1%) … Wikipedia
Factsheet
Standard Unicode Standard
Extends ASCII
Factsheet
Standard Unicode Standard
Extends ASCII
🌐
Wikipedia
en.wikipedia.org › wiki › UTF-8
UTF-8 - Wikipedia
2 weeks ago - That UTF-8 Clean-8 variant, implemented by Raku, is an encoder/decoder that preserves bytes as is (even illegal UTF-8 sequences) and allows for Normal Form Grapheme synthetics. Version 3 of the Python programming language treats each byte of an invalid UTF-8 bytestream as an error (see also ...
Top answer
1 of 2
65

You don't need to encode data that is already encoded. When you try to do that, Python will first try to decode it to unicode before it can encode it back to UTF-8. That is what is failing here:

>>> data = u'\u00c3'            # Unicode data
>>> data = data.encode('utf8')  # encoded to UTF-8
>>> data
'\xc3\x83'
>>> data.encode('utf8')         # Try to *re*-encode it
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Just write your data directly to the file, there is no need to encode already-encoded data.

If you instead build up unicode values instead, you would indeed have to encode those to be writable to a file. You'd want to use codecs.open() instead, which returns a file object that will encode unicode values to UTF-8 for you.

You also really don't want to write out the UTF-8 BOM, unless you have to support Microsoft tools that cannot read UTF-8 otherwise (such as MS Notepad).

For your MySQL insert problem, you need to do two things:

  • Add charset='utf8' to your MySQLdb.connect() call.

  • Use unicode objects, not str objects when querying or inserting, but use sql parameters so the MySQL connector can do the right thing for you:

    artiste = artiste.decode('utf8')  # it is already UTF8, decode to unicode
    
    c.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
    
    # ...
    
    c.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
    

It may actually work better if you used codecs.open() to decode the contents automatically instead:

import codecs

sql = mdb.connect('localhost','admin','ugo&(-@F','music_vibration', charset='utf8')

with codecs.open('config/index/'+index, 'r', 'utf8') as findex:
    for line in findex:
        if u'#artiste' not in line:
            continue

        artiste=line.split(u'[:::]')[1].strip()

    cursor = sql.cursor()
    cursor.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
    if not cursor.fetchone()[0]:
        cursor = sql.cursor()
        cursor.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
        artists_inserted += 1

You may want to brush up on Unicode and UTF-8 and encodings. I can recommend the following articles:

  • The Python Unicode HOWTO

  • Pragmatic Unicode by Ned Batchelder

  • The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

2 of 2
2

Unfortunately, the string.encode() method is not always reliable. Check out this thread for more information: What is the fool proof way to convert some string (utf-8 or else) to a simple ASCII string in python

Find elsewhere
🌐
Real Python
realpython.com › python-encodings-guide
Unicode & Character Encodings in Python: A Painless Guide – Real Python
May 20, 2019 - Python 3 source code is assumed to be UTF-8 by default. This means that you don’t need # -*- coding: UTF-8 -*- at the top of .py files in Python 3. All text (str) is Unicode by default. Encoded Unicode text is represented as binary data (bytes).
🌐
Medium
lynn-kwong.medium.com › understand-the-encoding-decoding-of-python-strings-unicode-utf-8-f6f97a909ee0
Understand the encoding/decoding of Python strings (Unicode/UTF-8) | by Lynn G. Kwong | Medium
August 25, 2023 - Python · Unicode · Utf 8 · Encoding · Hashing · Lynn G. Kwong · 5 min read · ·Dec 23, 2021 · -- Listen · Share · String is a common data type in Python and is used by us every day. In this article, the basics of the encoding/decoding of strings will be introduced which can clear your confusion in this field.
🌐
SSOJet
ssojet.com › character-encoding-decoding › utf-8-in-python
UTF-8 in Python | Encoding Standards for Programming Languages
UTF-8 is a variable-length encoding, meaning a single character might be represented by one to four bytes. When you encounter raw byte data, perhaps from a network socket or a binary file, you’ll need to decode it into a Python string to work with it as text.
Top answer
1 of 14
914

Rather than mess with .encode and .decode, specify the encoding when opening the file. The io module, added in Python 2.6, provides an io.open function, which allows specifying the file's encoding.

Supposing the file is encoded in UTF-8, we can use:

>>> import io
>>> f = io.open("test", mode="r", encoding="utf-8")

Then f.read returns a decoded Unicode object:

>>> f.read()
u'Capit\xe1l\n\n'

In 3.x, the io.open function is an alias for the built-in open function, which supports the encoding argument (it does not in 2.x).

We can also use open from the codecs standard library module:

>>> import codecs
>>> f = codecs.open("test", "r", "utf-8")
>>> f.read()
u'Capit\xe1l\n\n'

Note, however, that this can cause problems when mixing read() and readline().

2 of 14
126

In the notation u'Capit\xe1n\n' (should be just 'Capit\xe1n\n' in 3.x, and must be in 3.0 and 3.1), the \xe1 represents just one character. \x is an escape sequence, indicating that e1 is in hexadecimal.

Writing Capit\xc3\xa1n into the file in a text editor means that it actually contains \xc3\xa1. Those are 8 bytes and the code reads them all. We can see this by displaying the result:

# Python 3.x - reading the file as bytes rather than text,
# to ensure we see the raw data
>>> open('f2', 'rb').read()
b'Capit\\xc3\\xa1n\n'

# Python 2.x
>>> open('f2').read()
'Capit\\xc3\\xa1n\n'

Instead, just input characters like á in the editor, which should then handle the conversion to UTF-8 and save it.

In 2.x, a string that actually contains these backslash-escape sequences can be decoded using the string_escape codec:

# Python 2.x
>>> print 'Capit\\xc3\\xa1n\n'.decode('string_escape')
Capitán

The result is a str that is encoded in UTF-8 where the accented character is represented by the two bytes that were written \\xc3\\xa1 in the original string. To get a unicode result, decode again with UTF-8.

In 3.x, the string_escape codec is replaced with unicode_escape, and it is strictly enforced that we can only encode from a str to bytes, and decode from bytes to str. unicode_escape needs to start with a bytes in order to process the escape sequences (the other way around, it adds them); and then it will treat the resulting \xc3 and \xa1 as character escapes rather than byte escapes. As a result, we have to do a bit more work:

# Python 3.x
>>> 'Capit\\xc3\\xa1n\n'.encode('ascii').decode('unicode_escape').encode('latin-1').decode('utf-8')
'Capitán\n'
🌐
Python
docs.python.org › 2 › howto › unicode.html
Unicode HOWTO — Python 2.7.18 documentation
UTF-8 is one of the most commonly used encodings. UTF stands for “Unicode Transformation Format”, and the ‘8’ means that 8-bit numbers are used in the encoding.
🌐
Python Basics
python-basics-tutorial.readthedocs.io › en › latest › types › strings › encodings.html
Unicode and character encodings - Python Basics
8-bit · While Unicode is an abstract encoding standard, UTF-8 is a concrete encoding scheme. The Unicode standard is a mapping of characters to code points and defines several different encodings from a single character set.
🌐
Python Morsels
pythonmorsels.com › unicode-character-encodings-in-python
Unicode character encodings - Python Morsels
May 2, 2022 - But on Windows, the default character encoding is usually cp1252. Note: Since Python 3.6, all files are read and written by Python using utf-8 by default, even on Windows.
🌐
W3Schools
w3schools.com › python › ref_string_encode.asp
Python String encode() Method
Python Examples Python Compiler ... The encode() method encodes the string, using the specified encoding. If no encoding is specified, UTF-8 will be used....
🌐
Python Basics
python-basics-tutorial.readthedocs.io › en › 24.3.0 › appendix › encodings.html
Unicode and character encodings - Python Basics 24.3.0
8-bit · While Unicode is an abstract encoding standard, UTF-8 is a concrete encoding scheme. The Unicode standard is a mapping of characters to code points and defines several different encodings from a single character set.
🌐
Towards Data Science
towardsdatascience.com › a-guide-to-unicode-utf-8-and-strings-in-python-757a232db95c
A Guide to Unicode, UTF-8 and Strings in Python | by Sanket Gupta | Towards Data Science
September 24, 2024 - UTF-8: It uses 1, 2, 3 or 4 bytes to encode every code point. It is backwards compatible with ASCII. All English characters just need 1 byte — which is quite efficient. We only need more bytes if we are sending non-English characters.
🌐
Real Python
realpython.com › lessons › encoding-utf8
Encoding UTF-8 (Video) – Real Python
Well, there’s only 2 bits left and because all the bits have been used up, you know you’re done, which means it’s going to be 2 bytes, so use the 2-byte marker. 05:45 Finally, fill in the middle with some padding. This is the end result of the encoding. The left-hand side turns into C3, the right-hand side into A9. 05:57 If you remember from the session in the REPL, letter—having the code point E9—encoded into \xc3\xa9 (c3 a9). 06:06 So, this is how UTF-8 represents its information.
Published: June 30, 2020
🌐
Programiz
programiz.com › python-programming › methods › string › encode
Python String encode()
Online Python Online JavaScript Online SQL Online Java Online HTML Online C Online C++ Online C# Online PHP Online Swift Online Kotlin Online TypeScript Online Go Online Rust Online Scala Online Dart Online R Online Ruby ... The encode() method returns an encoded version of the given string. ... By default, the encode() method doesn't require any parameters. It returns an utf-8 encoded version of the string.