As mentioned in the comments, your question isn't very specific, so I'll try to give you some hints about character encodings, see if you can apply those to your specific case!
Unicode and Encoding
Here's a small primer about encoding. Basically, there are two ways to represent text in Python:
unicode. You can consider thatunicodeis the ultimate encoding, you should strive to use it everywhere. In Python 2.x source files,unicodestrings look likeu'some unicode'.str. This is encoded text - to be able to read it, you need to know the encoding (or guess it). In Python 2.x, those strings look like'some str'.
This changed in Python 3 (unicode is now str and str is now bytes).
How does that play out?
Usually, it's pretty straightforward to ensure that you code uses unicode for its execution, and uses str for I/O:
- Everything you receive is encoded, so you do
input_string.decode('encoding')to convert it tounicode. - Everything you need to output is unicode but needs to be encoded, so you do
output_string.encode('encoding').
The most common encodings are cp-1252 on Windows (on US or EU systems), and utf-8 on Linux.
Applying this to your case
I DO have to write äöü in a path, or it will not work
Windows natively uses unicode for file paths and names, so you should actually always use unicode for those.
It DOES have to be an ANSI-"encoded" file, or it will not work
When you write to the file, be sure to always run your output through output.encode('cp1252') (or whatever encoding ANSI would be on your system).
Things like line.write(str.decode('utf-8')) break the funktion of the file
By now you probably realized that:
- If
stras indeed anstrinstance, Python will try to convert it tounicodeusing theutf-8encoding, but then try to encode it again (likely inascii) to write it to the file - If
stris actually anunicodeinstance, Python will first encode it (likely inascii, and that will probably crash) to then be able to decode it.
Bottom line is, you need to know if str is unicode, you should encode it. If it's already encoded, don't touch it (or decode it then encode it if the encoding is not the one you want!).
A magical comment at the beginning of the script like # -- coding: iso-8859-1 -- does nothing here (though it is helpful when it comes to the mentioned Metadata and allowed characters in it...)
Not a surprise, this only tells Python what encoding should be used to read your source file so that non-ascii characters are properly recognized.
Oh, and i'm using Python 2.7.3. Third-Party modules dependencies, you know...
Python 3 probably is a big update in terms of unicode and encoding, but that doesn't mean Python 2.x can't make it work!
Will that solve your issue?
You can't be sure, it's possible that the problem lies in the player you're using, not in your code.
Once you output it, you should make sure that your script's output is readable using reference tools (such as Windows Explorer). If it is, but the player still can't open it, you should consider updating to a newer version.
Answer from Thomas Orozco on Stack OverflowAs mentioned in the comments, your question isn't very specific, so I'll try to give you some hints about character encodings, see if you can apply those to your specific case!
Unicode and Encoding
Here's a small primer about encoding. Basically, there are two ways to represent text in Python:
unicode. You can consider thatunicodeis the ultimate encoding, you should strive to use it everywhere. In Python 2.x source files,unicodestrings look likeu'some unicode'.str. This is encoded text - to be able to read it, you need to know the encoding (or guess it). In Python 2.x, those strings look like'some str'.
This changed in Python 3 (unicode is now str and str is now bytes).
How does that play out?
Usually, it's pretty straightforward to ensure that you code uses unicode for its execution, and uses str for I/O:
- Everything you receive is encoded, so you do
input_string.decode('encoding')to convert it tounicode. - Everything you need to output is unicode but needs to be encoded, so you do
output_string.encode('encoding').
The most common encodings are cp-1252 on Windows (on US or EU systems), and utf-8 on Linux.
Applying this to your case
I DO have to write äöü in a path, or it will not work
Windows natively uses unicode for file paths and names, so you should actually always use unicode for those.
It DOES have to be an ANSI-"encoded" file, or it will not work
When you write to the file, be sure to always run your output through output.encode('cp1252') (or whatever encoding ANSI would be on your system).
Things like line.write(str.decode('utf-8')) break the funktion of the file
By now you probably realized that:
- If
stras indeed anstrinstance, Python will try to convert it tounicodeusing theutf-8encoding, but then try to encode it again (likely inascii) to write it to the file - If
stris actually anunicodeinstance, Python will first encode it (likely inascii, and that will probably crash) to then be able to decode it.
Bottom line is, you need to know if str is unicode, you should encode it. If it's already encoded, don't touch it (or decode it then encode it if the encoding is not the one you want!).
A magical comment at the beginning of the script like # -- coding: iso-8859-1 -- does nothing here (though it is helpful when it comes to the mentioned Metadata and allowed characters in it...)
Not a surprise, this only tells Python what encoding should be used to read your source file so that non-ascii characters are properly recognized.
Oh, and i'm using Python 2.7.3. Third-Party modules dependencies, you know...
Python 3 probably is a big update in terms of unicode and encoding, but that doesn't mean Python 2.x can't make it work!
Will that solve your issue?
You can't be sure, it's possible that the problem lies in the player you're using, not in your code.
Once you output it, you should make sure that your script's output is readable using reference tools (such as Windows Explorer). If it is, but the player still can't open it, you should consider updating to a newer version.
On Windows there is special encoding available called mbcs, it converts between current default ANSI codepage and UNICODE. For example on a Spanish Language PC:
u'ñ'.encode('mbcs') -> '\xf1'
'\xf1'.decode('mbcs') -> u'ñ'
On Windows ANSI means current default multi-byte code page. For western European languages Windows ISO-8859-1, for eastern European languages windows ISO-8859-2) encoded byte string and other encodings for other languages as appropriate.
More info available at:
https://docs.python.org/2.4/lib/standard-encodings.html
See also:
https://docs.python.org/2/library/sys.html#sys.getfilesystemencoding
There is no "ANSI"-encoding. "ANSI" means "whatever the default single-byte encoding happens to be on your machine" – the term "ANSI" is inherently ambiguous. This means you must specify an actual encoding when reading the file.
For Windows machines in the Western Europe region, "ANSI" typically refers to Windows-1252. Other regions differ, but also your machine configuration might be different.
Python refers to Windows-1252 as cp1252. If that really is the encoding your file is in depends on the file itself, and can only be found out by looking at it.
Often text editors (not Notepad, real text editors) have an option to interpret a file in various encodings. Pick the one that makes the data look right (pay attention to accented characters) and then find out Python's name for it.
ANSI is not actually an encoding, you probably mean Windows-1252, which Python supports as 'cp1252'.
» pip install ansipants
Regarding this definition of ANSI encoding:
ANSI encoding is a slightly generic term used to refer to the standard code page on a system, usually Windows. It is more properly referred to as Windows-1252 on Western/U.S. systems.
You can try:
"\x98".encode("cp1252")
But, Python doesn’t allow it so, you can use ISO Latin1 instead:
"\x98".encode("iso_8859_1")
# b'\x98'
I'm not sure what you're expecting this to do. U+0098 is an unprintable character. You can try bytes([ord("\x98")]) to do a raw translation.
There's no ansi encoding in Python Standard Encodings.
Choose appropriate encodings from following link: Standard Encodings
OK,I find the answer.Thanks to @falsetru
#coding:utf-8
import chardet
def convertEncoding(from_encode,to_encode,old_filepath,target_file):
f1=file(old_filepath)
content2=[]
while True:
line=f1.readline()
content2.append(line.decode(from_encode).encode(to_encode))
if len(line) ==0:
break
f1.close()
f2=file(target_file,'w')
f2.writelines(content2)
f2.close()
convertFile = open('1234.csv','r')
data = convertFile.read()
convertFile.close()
convertEncoding(chardet.detect(data)['encoding'], "utf-8", "1234.csv", "1234_bak.csv")
Try this
#read input file
with codecs.open('USERS.CSV', 'r', encoding = 'latin-1') as file:
lines = file.read()
#write output file
with codecs.open('1_UserPython.CSV', 'w', encoding = 'utf_8_sig') as file:
file.write(lines)
To convert a file from utf8 to cp1252:
import io
with io.open(src_path, mode="r", encoding="utf8") as fd:
content = fd.read()
with io.open(dst_path, mode="w", encoding="cp1252") as fd:
fd.write(content)
Why don't you read the file and write it as UTF-8? You can do that in Python.
#to support encodings
import codecs
#read input file
with codecs.open(path, 'r', encoding = 'utf8') as file:
lines = file.read()
#write output file
with codecs.open(path, 'w', encoding = 'utf8') as file:
file.write(lines)
I appreciate that this is an old question but having just resolved a similar problem recently I thought I would share my solution.
I had a file being prepared by one program that I needed to import in to an sqlite3 database but the text file was always 'ANSI' and sqlite3 requires UTF-8.
The ANSI encoding is recognised as 'mbcs' in python and therefore the code I have used, ripping off something else I found is:
blockSize = 1048576
with codecs.open("your ANSI source file.txt","r",encoding="mbcs") as sourceFile:
with codecs.open("Your UTF-8 output file.txt","w",encoding="UTF-8") as targetFile:
while True:
contents = sourceFile.read(blockSize)
if not contents:
break
targetFile.write(contents)
The below link contains some information on the encoding types that I found on my research
https://docs.python.org/2.4/lib/standard-encodings.html
ANSI is not a character encoding (in common parlance it refers to certain escape sequences, though it's of course the acronym for the American National Standard Institute). You can set the encoding in Notepad++ (and check what encoding you're using) -- hopefully utf-8, because that's a universal encoding (lets you represent any Unicode point). You build unicode from your utf-8 encoded text with an explicit decode method call, or you read the file as unicode with a codecs.open (both require you to specify your encoding name -- again, hopefully 'utf8').
What does ASCII got to do with all of these?
Python has no way to find out what encoding was used to store text, so it assumes ascii by default. However, ASCII defines only first 128 chars, so anything outside results in decode error (which is actually good thing, since it does not let you use incorrectly decoded strings around).
Most of the time your string would be in utf-8, since its most common way to encode Unicode, so its usually safe to do s.decode('utf-8') on str type strings (or use unicode(s, 'utf-8') call)
If you dont know in advance what kind of encoding text has, and it provides no encoding metadata, you can try using chardet module.
BeautifulSoup can output result in different encodings and ways, so you just need to specify that you want unicode there.