As mentioned in the comments, your question isn't very specific, so I'll try to give you some hints about character encodings, see if you can apply those to your specific case!
Unicode and Encoding
Here's a small primer about encoding. Basically, there are two ways to represent text in Python:
unicode. You can consider thatunicodeis the ultimate encoding, you should strive to use it everywhere. In Python 2.x source files,unicodestrings look likeu'some unicode'.str. This is encoded text - to be able to read it, you need to know the encoding (or guess it). In Python 2.x, those strings look like'some str'.
This changed in Python 3 (unicode is now str and str is now bytes).
How does that play out?
Usually, it's pretty straightforward to ensure that you code uses unicode for its execution, and uses str for I/O:
- Everything you receive is encoded, so you do
input_string.decode('encoding')to convert it tounicode. - Everything you need to output is unicode but needs to be encoded, so you do
output_string.encode('encoding').
The most common encodings are cp-1252 on Windows (on US or EU systems), and utf-8 on Linux.
Applying this to your case
I DO have to write äöü in a path, or it will not work
Windows natively uses unicode for file paths and names, so you should actually always use unicode for those.
It DOES have to be an ANSI-"encoded" file, or it will not work
When you write to the file, be sure to always run your output through output.encode('cp1252') (or whatever encoding ANSI would be on your system).
Things like line.write(str.decode('utf-8')) break the funktion of the file
By now you probably realized that:
- If
stras indeed anstrinstance, Python will try to convert it tounicodeusing theutf-8encoding, but then try to encode it again (likely inascii) to write it to the file - If
stris actually anunicodeinstance, Python will first encode it (likely inascii, and that will probably crash) to then be able to decode it.
Bottom line is, you need to know if str is unicode, you should encode it. If it's already encoded, don't touch it (or decode it then encode it if the encoding is not the one you want!).
A magical comment at the beginning of the script like # -- coding: iso-8859-1 -- does nothing here (though it is helpful when it comes to the mentioned Metadata and allowed characters in it...)
Not a surprise, this only tells Python what encoding should be used to read your source file so that non-ascii characters are properly recognized.
Oh, and i'm using Python 2.7.3. Third-Party modules dependencies, you know...
Python 3 probably is a big update in terms of unicode and encoding, but that doesn't mean Python 2.x can't make it work!
Will that solve your issue?
You can't be sure, it's possible that the problem lies in the player you're using, not in your code.
Once you output it, you should make sure that your script's output is readable using reference tools (such as Windows Explorer). If it is, but the player still can't open it, you should consider updating to a newer version.
Answer from Thomas Orozco on Stack OverflowAs mentioned in the comments, your question isn't very specific, so I'll try to give you some hints about character encodings, see if you can apply those to your specific case!
Unicode and Encoding
Here's a small primer about encoding. Basically, there are two ways to represent text in Python:
unicode. You can consider thatunicodeis the ultimate encoding, you should strive to use it everywhere. In Python 2.x source files,unicodestrings look likeu'some unicode'.str. This is encoded text - to be able to read it, you need to know the encoding (or guess it). In Python 2.x, those strings look like'some str'.
This changed in Python 3 (unicode is now str and str is now bytes).
How does that play out?
Usually, it's pretty straightforward to ensure that you code uses unicode for its execution, and uses str for I/O:
- Everything you receive is encoded, so you do
input_string.decode('encoding')to convert it tounicode. - Everything you need to output is unicode but needs to be encoded, so you do
output_string.encode('encoding').
The most common encodings are cp-1252 on Windows (on US or EU systems), and utf-8 on Linux.
Applying this to your case
I DO have to write äöü in a path, or it will not work
Windows natively uses unicode for file paths and names, so you should actually always use unicode for those.
It DOES have to be an ANSI-"encoded" file, or it will not work
When you write to the file, be sure to always run your output through output.encode('cp1252') (or whatever encoding ANSI would be on your system).
Things like line.write(str.decode('utf-8')) break the funktion of the file
By now you probably realized that:
- If
stras indeed anstrinstance, Python will try to convert it tounicodeusing theutf-8encoding, but then try to encode it again (likely inascii) to write it to the file - If
stris actually anunicodeinstance, Python will first encode it (likely inascii, and that will probably crash) to then be able to decode it.
Bottom line is, you need to know if str is unicode, you should encode it. If it's already encoded, don't touch it (or decode it then encode it if the encoding is not the one you want!).
A magical comment at the beginning of the script like # -- coding: iso-8859-1 -- does nothing here (though it is helpful when it comes to the mentioned Metadata and allowed characters in it...)
Not a surprise, this only tells Python what encoding should be used to read your source file so that non-ascii characters are properly recognized.
Oh, and i'm using Python 2.7.3. Third-Party modules dependencies, you know...
Python 3 probably is a big update in terms of unicode and encoding, but that doesn't mean Python 2.x can't make it work!
Will that solve your issue?
You can't be sure, it's possible that the problem lies in the player you're using, not in your code.
Once you output it, you should make sure that your script's output is readable using reference tools (such as Windows Explorer). If it is, but the player still can't open it, you should consider updating to a newer version.
On Windows there is special encoding available called mbcs, it converts between current default ANSI codepage and UNICODE. For example on a Spanish Language PC:
u'ñ'.encode('mbcs') -> '\xf1'
'\xf1'.decode('mbcs') -> u'ñ'
On Windows ANSI means current default multi-byte code page. For western European languages Windows ISO-8859-1, for eastern European languages windows ISO-8859-2) encoded byte string and other encodings for other languages as appropriate.
More info available at:
https://docs.python.org/2.4/lib/standard-encodings.html
See also:
https://docs.python.org/2/library/sys.html#sys.getfilesystemencoding
encode - ANSI encoding in python - Stack Overflow
UTF-8 and ANSI encoding issue
Python: Multiple files ANSI to utf-8 converter | Notepad++ Community
How to set encoding as 'ANSI' using Python? - Stack Overflow
Regarding this definition of ANSI encoding:
ANSI encoding is a slightly generic term used to refer to the standard code page on a system, usually Windows. It is more properly referred to as Windows-1252 on Western/U.S. systems.
You can try:
"\x98".encode("cp1252")
But, Python doesn’t allow it so, you can use ISO Latin1 instead:
"\x98".encode("iso_8859_1")
# b'\x98'
I'm not sure what you're expecting this to do. U+0098 is an unprintable character. You can try bytes([ord("\x98")]) to do a raw translation.
» pip install ansipants
There is no "ANSI"-encoding. "ANSI" means "whatever the default single-byte encoding happens to be on your machine" – the term "ANSI" is inherently ambiguous. This means you must specify an actual encoding when reading the file.
For Windows machines in the Western Europe region, "ANSI" typically refers to Windows-1252. Other regions differ, but also your machine configuration might be different.
Python refers to Windows-1252 as cp1252. If that really is the encoding your file is in depends on the file itself, and can only be found out by looking at it.
Often text editors (not Notepad, real text editors) have an option to interpret a file in various encodings. Pick the one that makes the data look right (pay attention to accented characters) and then find out Python's name for it.
ANSI is not actually an encoding, you probably mean Windows-1252, which Python supports as 'cp1252'.
» pip install ansiparser
ANSI is not a character encoding (in common parlance it refers to certain escape sequences, though it's of course the acronym for the American National Standard Institute). You can set the encoding in Notepad++ (and check what encoding you're using) -- hopefully utf-8, because that's a universal encoding (lets you represent any Unicode point). You build unicode from your utf-8 encoded text with an explicit decode method call, or you read the file as unicode with a codecs.open (both require you to specify your encoding name -- again, hopefully 'utf8').
What does ASCII got to do with all of these?
Python has no way to find out what encoding was used to store text, so it assumes ascii by default. However, ASCII defines only first 128 chars, so anything outside results in decode error (which is actually good thing, since it does not let you use incorrectly decoded strings around).
Most of the time your string would be in utf-8, since its most common way to encode Unicode, so its usually safe to do s.decode('utf-8') on str type strings (or use unicode(s, 'utf-8') call)
If you dont know in advance what kind of encoding text has, and it provides no encoding metadata, you can try using chardet module.
BeautifulSoup can output result in different encodings and ways, so you just need to specify that you want unicode there.
MS Notepad gives the user a choice of 4 encodings, expressed in clumsy confusing terminology:
"Unicode" is UTF-16, written little-endian. "Unicode big endian" is UTF-16, written big-endian. In both UTF-16 cases, this means that the appropriate BOM will be written. Use utf-16 to decode such a file.
"UTF-8" is UTF-8; Notepad explicitly writes a "UTF-8 BOM". Use utf-8-sig to decode such a file.
"ANSI" is a shocker. This is MS terminology for "whatever the default legacy encoding is on this computer".
Here is a list of Windows encodings that I know of and the languages/scripts that they are used for:
cp874 Thai
cp932 Japanese
cp936 Unified Chinese (P.R. China, Singapore)
cp949 Korean
cp950 Traditional Chinese (Taiwan, Hong Kong, Macao(?))
cp1250 Central and Eastern Europe
cp1251 Cyrillic ( Belarusian, Bulgarian, Macedonian, Russian, Serbian, Ukrainian)
cp1252 Western European languages
cp1253 Greek
cp1254 Turkish
cp1255 Hebrew
cp1256 Arabic script
cp1257 Baltic languages
cp1258 Vietnamese
cp???? languages/scripts of India
If the file has been created on the computer where it is being read, then you can obtain the "ANSI" encoding by locale.getpreferredencoding(). Otherwise if you know where it came from, you can specify what encoding to use if it's not UTF-16. Failing that, guess.
Be careful using codecs.open() to read files on Windows. The docs say: """Note
Files are always opened in binary mode, even if no binary mode was specified. This is done to avoid data loss due to encodings using 8-bit values. This means that no automatic conversion of '\n' is done on reading and writing.""" This means that your lines will end in \r\n and you will need/want to strip those off.
Putting it all together:
Sample text file, saved with all 4 encoding choices, looks like this in Notepad:
The quick brown fox jumped over the lazy dogs.
àáâãäå
Here is some demo code:
import locale
def guess_notepad_encoding(filepath, default_ansi_encoding=None):
with open(filepath, 'rb') as f:
data = f.read(3)
if data[:2] in ('\xff\xfe', '\xfe\xff'):
return 'utf-16'
if data == u''.encode('utf-8-sig'):
return 'utf-8-sig'
# presumably "ANSI"
return default_ansi_encoding or locale.getpreferredencoding()
if __name__ == "__main__":
import sys, glob, codecs
defenc = sys.argv[1]
for fpath in glob.glob(sys.argv[2]):
print
print (fpath, defenc)
with open(fpath, 'rb') as f:
print "raw:", repr(f.read())
enc = guess_notepad_encoding(fpath, defenc)
print "guessed encoding:", enc
with codecs.open(fpath, 'r', enc) as f:
for lino, line in enumerate(f, 1):
print lino, repr(line)
print lino, repr(line.rstrip('\r\n'))
and here is the output when run in a Windows "Command Prompt" window using the command \python27\python read_notepad.py "" t1-*.txt
('t1-ansi.txt', '')
raw: 'The quick brown fox jumped over the lazy dogs.\r\n\xe0\xe1\xe2\xe3\xe4\xe5
\r\n'
guessed encoding: cp1252
1 u'The quick brown fox jumped over the lazy dogs.\r\n'
1 u'The quick brown fox jumped over the lazy dogs.'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5\r\n'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5'
('t1-u8.txt', '')
raw: '\xef\xbb\xbfThe quick brown fox jumped over the lazy dogs.\r\n\xc3\xa0\xc3
\xa1\xc3\xa2\xc3\xa3\xc3\xa4\xc3\xa5\r\n'
guessed encoding: utf-8-sig
1 u'The quick brown fox jumped over the lazy dogs.\r\n'
1 u'The quick brown fox jumped over the lazy dogs.'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5\r\n'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5'
('t1-uc.txt', '')
raw: '\xff\xfeT\x00h\x00e\x00 \x00q\x00u\x00i\x00c\x00k\x00 \x00b\x00r\x00o\x00w
\x00n\x00 \x00f\x00o\x00x\x00 \x00j\x00u\x00m\x00p\x00e\x00d\x00 \x00o\x00v\x00e
\x00r\x00 \x00t\x00h\x00e\x00 \x00l\x00a\x00z\x00y\x00 \x00d\x00o\x00g\x00s\x00.
\x00\r\x00\n\x00\xe0\x00\xe1\x00\xe2\x00\xe3\x00\xe4\x00\xe5\x00\r\x00\n\x00'
guessed encoding: utf-16
1 u'The quick brown fox jumped over the lazy dogs.\r\n'
1 u'The quick brown fox jumped over the lazy dogs.'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5\r\n'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5'
('t1-ucb.txt', '')
raw: '\xfe\xff\x00T\x00h\x00e\x00 \x00q\x00u\x00i\x00c\x00k\x00 \x00b\x00r\x00o\
x00w\x00n\x00 \x00f\x00o\x00x\x00 \x00j\x00u\x00m\x00p\x00e\x00d\x00 \x00o\x00v\
x00e\x00r\x00 \x00t\x00h\x00e\x00 \x00l\x00a\x00z\x00y\x00 \x00d\x00o\x00g\x00s\
x00.\x00\r\x00\n\x00\xe0\x00\xe1\x00\xe2\x00\xe3\x00\xe4\x00\xe5\x00\r\x00\n'
guessed encoding: utf-16
1 u'The quick brown fox jumped over the lazy dogs.\r\n'
1 u'The quick brown fox jumped over the lazy dogs.'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5\r\n'
2 u'\xe0\xe1\xe2\xe3\xe4\xe5'
Things to be aware of:
(1) "mbcs" is a file-system pseudo-encoding which has no relevance at all to decoding the contents of files. On a system where the default encoding is cp1252, it makes like latin1 (aarrgghh!!); see below
>>> all_bytes = "".join(map(chr, range(256)))
>>> u1 = all_bytes.decode('cp1252', 'replace')
>>> u2 = all_bytes.decode('mbcs', 'replace')
>>> u1 == u2
False
>>> [(i, u1[i], u2[i]) for i in xrange(256) if u1[i] != u2[i]]
[(129, u'\ufffd', u'\x81'), (141, u'\ufffd', u'\x8d'), (143, u'\ufffd', u'\x8f')
, (144, u'\ufffd', u'\x90'), (157, u'\ufffd', u'\x9d')]
>>>
(2) chardet is very good at detecting encodings based on non-Latin scripts (Chinese/Japanese/Korean, Cyrillic, Hebrew, Greek) but not much good at Latin-based encodings (Western/Central/Eastern Europe, Turkish, Vietnamese) and doesn't grok Arabic at all.
Notepad saves Unicode files with a byte order mark. This means that the first bytes of the file will be:
- EF BB BF -- UTF-8
- FF FE -- "Unicode" (actually UTF-16 little-endian, looks like)
- FE FF -- "Unicode big-endian" (looks like UTF-16 big-endian)
Other text editors may or may not have the same behavior, but if you know for sure Notepad is being used, this will give you a decent heuristic for auto-selecting the encoding. All these sequences are valid in the ANSI encoding as well, however, so it is possible for this heuristic to make mistakes. It is not possible to guarantee that the correct encoding is used.
ANSI means more or less nothing --- the most probable candidate for your encoding is Windows-1252. You can convert the file with
iconv -f WINDOWS-1252 -t utf8 < filein.txt > fileout.txt
but remember, most encodings (read the linked article if you are not sure about what that mean) can't be reliably guessed, so you need to know the exact encoding to give sense to your data.
From the comments it seems that you are most probably looking for some Arabic encoding --- in that case check WINDOWS-1256.
The list of available encodings for iconv is on github, or you can find it with the command
iconv --list
Notice that just at the start of the list there are a bunch of "ANSI"-like encodings.
If you don't want to convert it as mentioned in Rmano's answer, you may use Geany or Kate to read/edit them ar they are, default Arabic encoding in Windows is CP-1256 (WINDOWS-1256). Just keep in mind that switching to UTF-8 is recommended over all. It is supported by Notepad++ and Windows default note editor.
Geany
sudo apt-get install geany

Kate
sudo apt-get install kate

There's no ansi encoding in Python Standard Encodings.
Choose appropriate encodings from following link: Standard Encodings
OK,I find the answer.Thanks to @falsetru
#coding:utf-8
import chardet
def convertEncoding(from_encode,to_encode,old_filepath,target_file):
f1=file(old_filepath)
content2=[]
while True:
line=f1.readline()
content2.append(line.decode(from_encode).encode(to_encode))
if len(line) ==0:
break
f1.close()
f2=file(target_file,'w')
f2.writelines(content2)
f2.close()
convertFile = open('1234.csv','r')
data = convertFile.read()
convertFile.close()
convertEncoding(chardet.detect(data)['encoding'], "utf-8", "1234.csv", "1234_bak.csv")