Use chardet library. It is super easy
import chardet
the_encoding = chardet.detect('your string')['encoding']
and that's it!
in python3 you need to provide type bytes or bytearray so:
import chardet
the_encoding = chardet.detect(b'your string')['encoding']
Use chardet library. It is super easy
import chardet
the_encoding = chardet.detect('your string')['encoding']
and that's it!
in python3 you need to provide type bytes or bytearray so:
import chardet
the_encoding = chardet.detect(b'your string')['encoding']
if your files either in cp1252 and utf-8, then there is an easy way.
import logging
def force_decode(string, codecs=['utf8', 'cp1252']):
for i in codecs:
try:
return string.decode(i)
except UnicodeDecodeError:
pass
logging.warn("cannot decode url %s" % ([string]))
for item in os.listdir(rootPath):
#Convert to Unicode
if isinstance(item, str):
item = force_decode(item)
print item
otherwise, there is a charset detect lib.
Python - detect charset and convert to utf-8
https://pypi.python.org/pypi/chardet
EDIT: chardet seems to be unmantained but most of the answer applies. Check https://pypi.org/project/charset-normalizer/ for an alternative
Correctly detecting the encoding all times is impossible.
(From chardet FAQ:)
However, some encodings are optimized for specific languages, and languages are not random. Some character sequences pop up all the time, while other sequences make no sense. A person fluent in English who opens a newspaper and finds “txzqJv 2!dasd0a QqdKjvz” will instantly recognize that that isn't English (even though it is composed entirely of English letters). By studying lots of “typical” text, a computer algorithm can simulate this kind of fluency and make an educated guess about a text's language.
There is the chardet library that uses that study to try to detect encoding. chardet is a port of the auto-detection code in Mozilla.
You can also use UnicodeDammit. It will try the following methods:
- An encoding discovered in the document itself: for instance, in an XML declaration or (for HTML documents) an http-equiv META tag. If Beautiful Soup finds this kind of encoding within the document, it parses the document again from the beginning and gives the new encoding a try. The only exception is if you explicitly specified an encoding, and that encoding actually worked: then it will ignore any encoding it finds in the document.
- An encoding sniffed by looking at the first few bytes of the file. If an encoding is detected at this stage, it will be one of the UTF-* encodings, EBCDIC, or ASCII.
- An encoding sniffed by the chardet library, if you have it installed.
- UTF-8
- Windows-1252
Another option for working out the encoding is to use libmagic (which is the code behind the file command). There are a profusion of Python bindings available.
The Python bindings that live in the file source tree are available as the python-magic (or python3-magic) debian package. It can determine the encoding of a file by doing:
import magic
blob = open('unknown-file', 'rb').read()
m = magic.open(magic.MAGIC_MIME_ENCODING)
m.load()
encoding = m.buffer(blob) # "utf-8", "us-ascii", etc.
There is an identically named, but incompatible, python-magic pip package on PyPI that also uses libmagic. It can also get the encoding, by doing:
import magic
m = magic.Magic(mime_encoding=True)
encoding = m.from_file('unknown-file')
In Python 3, all strings are sequences of Unicode characters. There is a bytes type that holds raw bytes.
In Python 2, a string may be of type str or of type unicode. You can tell which using code something like this:
def whatisthis(s):
if isinstance(s, str):
print "ordinary string"
elif isinstance(s, unicode):
print "unicode string"
else:
print "not a string"
This does not distinguish "Unicode or ASCII"; it only distinguishes Python types. A Unicode string may consist of purely characters in the ASCII range, and a bytestring may contain ASCII, encoded Unicode, or even non-textual data.
How to tell if an object is a unicode string or a byte string
You can use type or isinstance.
In Python 2:
>>> type(u'abc') # Python 2 unicode string literal
<type 'unicode'>
>>> type('abc') # Python 2 byte string literal
<type 'str'>
In Python 2, str is just a sequence of bytes. Python doesn't know what
its encoding is. The unicode type is the safer way to store text.
If you want to understand this more, I recommend http://farmdev.com/talks/unicode/.
In Python 3:
>>> type('abc') # Python 3 unicode string literal
<class 'str'>
>>> type(b'abc') # Python 3 byte string literal
<class 'bytes'>
In Python 3, str is like Python 2's unicode, and is used to
store text. What was called str in Python 2 is called bytes in Python 3.
How to tell if a byte string is valid utf-8 or ascii
You can call decode. If it raises a UnicodeDecodeError exception, it wasn't valid.
>>> u_umlaut = b'\xc3\x9c' # UTF-8 representation of the letter 'Ü'
>>> u_umlaut.decode('utf-8')
u'\xdc'
>>> u_umlaut.decode('ascii')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
In my experience enca commandline tool is pretty good at guessing encoding correctly:
http://linux.die.net/man/1/enca
In Python, there's chardet:
https://github.com/chardet/chardet
I just think of a way, you can decode the string in every possible encodings.
The following encodings are borrowed from Python's Standard Encodings
code_list = ["ascii", "big5", "big5hkscs", "cp037", "cp424", "cp437", "cp500",
"cp720", "cp737", "cp775", "cp850", "cp852", "cp855", "cp856", "cp857", "cp858",
"cp860", "cp861", "cp862", "cp863", "cp864", "cp865", "cp866", "cp869", "cp874",
"cp875", "cp932", "cp949", "cp950", "cp1006", "cp1026", "cp1140", "cp1250", "cp1251",
"cp1252", "cp1253", "cp1254", "cp1255", "cp1256", "cp1257", "cp1258", "euc_jp",
"euc_jis_2004", "euc_jisx0213", "euc_kr", "gb2312", "gbk", "gb18030", "hz", "iso2022_jp",
"iso2022_jp_1", "iso2022_jp_2", "iso2022_jp_2004", "iso2022_jp_3", "iso2022_jp_ext",
"iso2022_kr", "latin_1", "iso8859_2", "iso8859_3", "iso8859_4", "iso8859_5", "iso8859_6",
"iso8859_7", "iso8859_8", "iso8859_9", "iso8859_10", "iso8859_13", "iso8859_14",
"iso8859_15", "iso8859_16", "johab", "koi8_r", "koi8_u", "mac_cyrillic", "mac_greek",
"mac_iceland", "mac_latin2", "mac_roman", "mac_turkish", "ptcp154", "shift_jis",
"shift_jis_2004", "shift_jisx0213", "utf_32", "utf_32_be", "utf_32_le", "utf_16",
"utf_16_be", "utf_16_le", "utf_7", "utf_8", "utf_8_sig", "idna", "mbcs", "palmos",
"punycode", "raw_unicode_escape", "rot_13", "undefined", "unicode_escape",
"unicode_internal", "base64_codec", "bz2_codec", "hex_codec", "quopri_codec",
"string_escape", "uu_codec", "zlib_codec"]
s = '\x86\x9cG<!\xd9F@\xb4\n\xd6\xd4(\x9cb\xfe'
for i in code_list:
try:
print 'Using {0} to decode......{1:<30}'.format(i,s.decode(i).encode('utf-8'))
except Exception as e:
# pass
print e
Unfortunately there is no 'correct' way to determine the encoding of a file by looking at the file itself. This is a universal problem, not limited to python or any particular file system.
If you're reading an XML file, the first line in the file might give you a hint of what the encoding is.
Otherwise, you will have to use some heuristics-based approach like chardet (one of the solutions given in other answers) which tries to guess the encoding by examining the data in the file in raw byte format. If you're on Windows, I believe the Windows API also exposes methods to try and guess the encoding based on the data in the file.
You may use BOM (http://en.wikipedia.org/wiki/Byte_order_mark) to detect the encoding, or try this library:
https://github.com/chardet/chardet