You don't need to encode data that is already encoded. When you try to do that, Python will first try to decode it to unicode before it can encode it back to UTF-8. That is what is failing here:

>>> data = u'\u00c3'            # Unicode data
>>> data = data.encode('utf8')  # encoded to UTF-8
>>> data
'\xc3\x83'
>>> data.encode('utf8')         # Try to *re*-encode it
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Just write your data directly to the file, there is no need to encode already-encoded data.

If you instead build up unicode values instead, you would indeed have to encode those to be writable to a file. You'd want to use codecs.open() instead, which returns a file object that will encode unicode values to UTF-8 for you.

You also really don't want to write out the UTF-8 BOM, unless you have to support Microsoft tools that cannot read UTF-8 otherwise (such as MS Notepad).

For your MySQL insert problem, you need to do two things:

  • Add charset='utf8' to your MySQLdb.connect() call.

  • Use unicode objects, not str objects when querying or inserting, but use sql parameters so the MySQL connector can do the right thing for you:

    artiste = artiste.decode('utf8')  # it is already UTF8, decode to unicode
    
    c.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
    
    # ...
    
    c.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
    

It may actually work better if you used codecs.open() to decode the contents automatically instead:

import codecs

sql = mdb.connect('localhost','admin','ugo&(-@F','music_vibration', charset='utf8')

with codecs.open('config/index/'+index, 'r', 'utf8') as findex:
    for line in findex:
        if u'#artiste' not in line:
            continue

        artiste=line.split(u'[:::]')[1].strip()

    cursor = sql.cursor()
    cursor.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
    if not cursor.fetchone()[0]:
        cursor = sql.cursor()
        cursor.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
        artists_inserted += 1

You may want to brush up on Unicode and UTF-8 and encodings. I can recommend the following articles:

  • The Python Unicode HOWTO

  • Pragmatic Unicode by Ned Batchelder

  • The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

Answer from Martijn Pieters on Stack Overflow
🌐
Python documentation
docs.python.org β€Ί 3 β€Ί howto β€Ί unicode.html
Unicode HOWTO β€” Python 3.14.7 documentation
You could then edit Python source ... characters used at runtime. Python supports writing source code in UTF-8 by default, but you can use almost any encoding if you declare the encoding being used....
🌐
W3Schools
w3schools.com β€Ί python β€Ί ref_string_encode.asp
Python String encode() Method
Python Examples Python Compiler ... The encode() method encodes the string, using the specified encoding. If no encoding is specified, UTF-8 will be used....
Discussions

unicode - python encoding utf-8 - Stack Overflow
I am doing some scripts in python. I create a string that I save in a file. This string got lot of data, coming from the arborescence and filenames of a directory. According to convmv, all my More on stackoverflow.com
🌐 stackoverflow.com
Writing files with UTF-8 encoding?
First, are you using Python 2 or 3? How Python 3 handles string encodings is one of the primary differences between the two. Basically, in your program you manipulate unicode strings. In Python 2 they will be unicode objects and the literals will have the u prefix. In python 3 they will be str objects. Before you write them to a file, the string needs to be encoded into a byte-representation. You do this with just mystring.encode("UTF-8"). The result of that will be a byte string, bytes in Python 3 or str in Python 2. If you try and write a unicode string to a file directly, it will probably use some system default encoding, or error depending on how you're opening the file and your python version. Also relevant: are you using some XML library to write the file? How are you getting the chinese characters into your program? Are you sure they're not gibberish before you even write them out? I don't have enough information to answer those questions for you. More on reddit.com
🌐 r/learnpython
7
6
May 13, 2013
Really confused about UTF-8
The json module uses ascii by default to make the data more robust to sending across the internet. When you load the data with the json module the data is unescaped and converted back to unicode. Edit to clarify the json module, not the json spec More on reddit.com
🌐 r/learnpython
13
9
July 1, 2022
OK, seriously. Why isn't UTF-8 the default encoding in Python?
Many many people asked the same thing. And hence Python 3 was born. and for other reasons as well but this is a big one UTF-8 is default in Python 3. More on reddit.com
🌐 r/learnpython
18
8
November 4, 2017
Top answer
1 of 2
65

You don't need to encode data that is already encoded. When you try to do that, Python will first try to decode it to unicode before it can encode it back to UTF-8. That is what is failing here:

>>> data = u'\u00c3'            # Unicode data
>>> data = data.encode('utf8')  # encoded to UTF-8
>>> data
'\xc3\x83'
>>> data.encode('utf8')         # Try to *re*-encode it
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Just write your data directly to the file, there is no need to encode already-encoded data.

If you instead build up unicode values instead, you would indeed have to encode those to be writable to a file. You'd want to use codecs.open() instead, which returns a file object that will encode unicode values to UTF-8 for you.

You also really don't want to write out the UTF-8 BOM, unless you have to support Microsoft tools that cannot read UTF-8 otherwise (such as MS Notepad).

For your MySQL insert problem, you need to do two things:

  • Add charset='utf8' to your MySQLdb.connect() call.

  • Use unicode objects, not str objects when querying or inserting, but use sql parameters so the MySQL connector can do the right thing for you:

    artiste = artiste.decode('utf8')  # it is already UTF8, decode to unicode
    
    c.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
    
    # ...
    
    c.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
    

It may actually work better if you used codecs.open() to decode the contents automatically instead:

import codecs

sql = mdb.connect('localhost','admin','ugo&(-@F','music_vibration', charset='utf8')

with codecs.open('config/index/'+index, 'r', 'utf8') as findex:
    for line in findex:
        if u'#artiste' not in line:
            continue

        artiste=line.split(u'[:::]')[1].strip()

    cursor = sql.cursor()
    cursor.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
    if not cursor.fetchone()[0]:
        cursor = sql.cursor()
        cursor.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
        artists_inserted += 1

You may want to brush up on Unicode and UTF-8 and encodings. I can recommend the following articles:

  • The Python Unicode HOWTO

  • Pragmatic Unicode by Ned Batchelder

  • The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

2 of 2
2

Unfortunately, the string.encode() method is not always reliable. Check out this thread for more information: What is the fool proof way to convert some string (utf-8 or else) to a simple ASCII string in python

🌐
Honeybadger
honeybadger.io β€Ί blog β€Ί python-character-encoding
Python developer's guide to character encoding - Honeybadger Developer Blog
March 6, 2023 - Python uses UTF-8 by default, which means it does not need to be specified in every Python file. To encode a string into bytes, add the encode method, which will return the binary representation of the string.
🌐
GeeksforGeeks
geeksforgeeks.org β€Ί python β€Ί convert-a-string-to-utf-8-in-python
Convert a String to Utf-8 in Python - GeeksforGeeks
July 23, 2025 - In this example, the encode method is called on the original_string with the argument 'utf-8'. The result is a bytes object containing the UTF-8 representation of the original string.
Find elsewhere
🌐
Programiz
programiz.com β€Ί python-programming β€Ί methods β€Ί string β€Ί encode
Python String encode()
Online Python Online JavaScript ... Online Scala Online Dart Online R Online Ruby ... The encode() method returns an encoded version of the given string. ... By default, the encode() method doesn't require any parameters. It returns an utf-8 encoded version of the ...
🌐
Python
docs.python.org β€Ί 3 β€Ί library β€Ί codecs.html
codecs β€” Codec registry and base classes
Creates a StreamRecoder instance which implements a two-way conversion: encode and decode work on the frontend β€” the data visible to code calling read() and write(), while Reader and Writer work on the backend β€” the data in stream. You can use these objects to do transparent transcodings, e.g., from Latin-1 to UTF-8 and back.
🌐
Python Morsels
pythonmorsels.com β€Ί unicode-character-encodings-in-python
Unicode character encodings - Python Morsels
May 2, 2022 - But on Windows, the default character encoding is usually cp1252. Note: Since Python 3.6, all files are read and written by Python using utf-8 by default, even on Windows.
🌐
Python Basics
python-basics-tutorial.readthedocs.io β€Ί en β€Ί latest β€Ί types β€Ί strings β€Ί encodings.html
Unicode and character encodings - Python Basics
Added in version 3.15: Python 3.15 uses UTF-8 as the default encoding, regardless of the system environment. This means that I/O operations without explicit encoding, for example open("EXAMPLE.TXT"), use UTF-8.
🌐
npm
npmjs.com β€Ί package β€Ί bcrypt
bcrypt - npm
If a string is provided, it will be encoded using UTF-8.
      Β» npm install bcrypt
    
Published: May 11, 2025
Version: 6.0.0
🌐
Codemia
codemia.io β€Ί home β€Ί knowledge hub β€Ί working with utf-8 encoding in python source
Working with UTF-8 encoding in Python source | Codemia
September 23, 2025 - The remaining confusion usually ... source code as Unicode code points. If your .py file is saved as UTF-8, you can write non-ASCII characters directly in string literals, comments, and even identifiers....
🌐
HTTPX
python-httpx.org
HTTPX
>>> import httpx >>> r = httpx.get('https://www.example.org/') >>> r <Response [200 OK]> >>> r.status_code 200 >>> r.headers['content-type'] 'text/html; charset=UTF-8' >>> r.text '<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'
🌐
Python
peps.python.org β€Ί pep-0686
PEP 686 – Make UTF-8 mode default | peps.python.org
March 18, 2022 - Inconsistent default encoding causes many bugs. Python will enable UTF-8 mode by default from Python 3.15.
🌐
Online Tools
emn178.github.io β€Ί online-tools β€Ί base64_decode.html
Base64 Decode - Online Tools
The standard Base64 value SGVsbG8= decodes to Hello in UTF-8. The bytes fbff encode as +/8= in standard Base64 and -_8= in Base64URL; both are accepted by this page’s Standard decoder.
🌐
Wikipedia
en.wikipedia.org β€Ί wiki β€Ί UTF-8
UTF-8 - Wikipedia
1 week ago - That UTF-8 Clean-8 variant, implemented by Raku, is an encoder/decoder that preserves bytes as is (even illegal UTF-8 sequences) and allows for Normal Form Grapheme synthetics. Version 3 of the Python programming language treats each byte of an invalid UTF-8 bytestream as an error (see also ...
🌐
MojoAuth
mojoauth.com β€Ί character-encoding-decoding β€Ί unicode-encoding--python
Unicode Encoding : Python | Encoding Solutions Across Programming Languages
When you encode a string in Python, you can specify the encoding format like this: # Example of encoding a string to UTF-8 text = "Hello, world! 🌍" encoded_text = text.encode('utf-8') # Convert to bytes print(encoded_text) # Output: b'Hello, ...
🌐
Pandas
pandas.pydata.org β€Ί docs β€Ί reference β€Ί api β€Ί pandas.DataFrame.to_csv.html
pandas.DataFrame.to_csv β€” pandas 3.0.6 documentation
A string representing the encoding to use in the output file, defaults to β€˜utf-8’. encoding is not supported if path_or_buf is a non-binary file object.
🌐
ntfy
docs.ntfy.sh
ntfy
requests.post("https://ntfy.sh/mytopic", data="Backup successful πŸ˜€".encode(encoding='utf-8'))