EDIT: chardet seems to be unmantained but most of the answer applies. Check https://pypi.org/project/charset-normalizer/ for an alternative
Correctly detecting the encoding all times is impossible.
(From chardet FAQ:)
However, some encodings are optimized for specific languages, and languages are not random. Some character sequences pop up all the time, while other sequences make no sense. A person fluent in English who opens a newspaper and finds “txzqJv 2!dasd0a QqdKjvz” will instantly recognize that that isn't English (even though it is composed entirely of English letters). By studying lots of “typical” text, a computer algorithm can simulate this kind of fluency and make an educated guess about a text's language.
There is the chardet library that uses that study to try to detect encoding. chardet is a port of the auto-detection code in Mozilla.
You can also use UnicodeDammit. It will try the following methods:
- An encoding discovered in the document itself: for instance, in an XML declaration or (for HTML documents) an http-equiv META tag. If Beautiful Soup finds this kind of encoding within the document, it parses the document again from the beginning and gives the new encoding a try. The only exception is if you explicitly specified an encoding, and that encoding actually worked: then it will ignore any encoding it finds in the document.
- An encoding sniffed by looking at the first few bytes of the file. If an encoding is detected at this stage, it will be one of the UTF-* encodings, EBCDIC, or ASCII.
- An encoding sniffed by the chardet library, if you have it installed.
- UTF-8
- Windows-1252
EDIT: chardet seems to be unmantained but most of the answer applies. Check https://pypi.org/project/charset-normalizer/ for an alternative
Correctly detecting the encoding all times is impossible.
(From chardet FAQ:)
However, some encodings are optimized for specific languages, and languages are not random. Some character sequences pop up all the time, while other sequences make no sense. A person fluent in English who opens a newspaper and finds “txzqJv 2!dasd0a QqdKjvz” will instantly recognize that that isn't English (even though it is composed entirely of English letters). By studying lots of “typical” text, a computer algorithm can simulate this kind of fluency and make an educated guess about a text's language.
There is the chardet library that uses that study to try to detect encoding. chardet is a port of the auto-detection code in Mozilla.
You can also use UnicodeDammit. It will try the following methods:
- An encoding discovered in the document itself: for instance, in an XML declaration or (for HTML documents) an http-equiv META tag. If Beautiful Soup finds this kind of encoding within the document, it parses the document again from the beginning and gives the new encoding a try. The only exception is if you explicitly specified an encoding, and that encoding actually worked: then it will ignore any encoding it finds in the document.
- An encoding sniffed by looking at the first few bytes of the file. If an encoding is detected at this stage, it will be one of the UTF-* encodings, EBCDIC, or ASCII.
- An encoding sniffed by the chardet library, if you have it installed.
- UTF-8
- Windows-1252
Another option for working out the encoding is to use libmagic (which is the code behind the file command). There are a profusion of Python bindings available.
The Python bindings that live in the file source tree are available as the python-magic (or python3-magic) debian package. It can determine the encoding of a file by doing:
import magic
blob = open('unknown-file', 'rb').read()
m = magic.open(magic.MAGIC_MIME_ENCODING)
m.load()
encoding = m.buffer(blob) # "utf-8", "us-ascii", etc.
There is an identically named, but incompatible, python-magic pip package on PyPI that also uses libmagic. It can also get the encoding, by doing:
import magic
m = magic.Magic(mime_encoding=True)
encoding = m.from_file('unknown-file')
How to know the encoding of a file in Python? - Stack Overflow
How to determine encoding of text file?
python - How do I detect if a file is encoded using UTF-8? - Stack Overflow
The use of open(encoding="utf-8")
Files generally indicate their encoding with a file header. There are many examples here. However, even reading the header you can never be sure what encoding a file is really using.
For example, a file with the first three bytes 0xEF,0xBB,0xBF is probably a UTF-8 encoded file. However, it might be an ISO-8859-1 file which happens to start with the characters . Or it might be a different file type entirely.
Notepad++ does its best to guess what encoding a file is using, and most of the time it gets it right. Sometimes it does get it wrong though - that's why that 'Encoding' menu is there, so you can override its best guess.
For the two encodings you mention:
- The "UCS-2 Little Endian" files are UTF-16 files (based on what I understand from the info here) so probably start with
0xFF,0xFEas the first 2 bytes. From what I can tell, Notepad++ describes them as "UCS-2" since it doesn't support certain facets of UTF-16. - The "UTF-8 without BOM" files don't have any header bytes. That's what the "without BOM" bit means.
You cannot. If you could do that, there would not be so many web sites or text files with “random gibberish” out there. That's why the encoding is usually sent along with the payload as meta data.
In case it's not, all you can do is a “smart guess” but the result is often ambiguous since the same byte sequence might be valid in several encodings.
Unfortunately there is no 'correct' way to determine the encoding of a file by looking at the file itself. This is a universal problem, not limited to python or any particular file system.
If you're reading an XML file, the first line in the file might give you a hint of what the encoding is.
Otherwise, you will have to use some heuristics-based approach like chardet (one of the solutions given in other answers) which tries to guess the encoding by examining the data in the file in raw byte format. If you're on Windows, I believe the Windows API also exposes methods to try and guess the encoding based on the data in the file.
You may use BOM (http://en.wikipedia.org/wiki/Byte_order_mark) to detect the encoding, or try this library:
https://github.com/chardet/chardet
I'm trying to open a file and read it line by line, however when I call readline() I get UnicodeDecodeError's such as this:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x96 in position 430: unexpected code bytes.
when trying to read this line:
The Last Answer by Isaac Asimov — © 1980
I'm guessing it's because of the ©, so I figured opening it with encoding='utf-8' would help, but I get the same error (except w/utf-8 codec instead obviously). So.. I'm assuming I just need to find out what this is encoded in, so how to do that? This is just something I copied and pasted off of the web into a plain txt file btw, not sure if that matters. It's the last answer if anyone's interested. Great read.
You mentioned in a comment you only need to detect UTF-8. If you know the alternative consists of only single byte encodings, then there is a solution that often works.
If you know it's either UTF-8 or single byte encoding like latin-1, then try opening it first in UTF-8 and then in the other encoding. If the file contains only ASCII characters, it will end up opened in UTF-8 even if it was intended as the other encoding. If it contains any non-ASCII characters, this will almost always correctly detect the right character set between the two.
try:
# or codecs.open on Python <= 2.5
# or io.open on Python > 2.5 and <= 2.7
filedata = open(filename, encoding='UTF-8').read()
except:
filedata = open(filename, encoding='other-single-byte-encoding').read()
Your best bet is to use the chardet package from PyPI, either directly or through UnicodeDamnit from BeautifulSoup:
chardet 1.0.1
Universal encoding detector
Detects:
- ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
- Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
- EUC-JP, SHIFT_JIS, ISO-2022-JP (Japanese)
- EUC-KR, ISO-2022-KR (Korean)
- KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
- ISO-8859-2, windows-1250 (Hungarian)
- ISO-8859-5, windows-1251 (Bulgarian)
- windows-1252 (English)
- ISO-8859-7, windows-1253 (Greek)
- ISO-8859-8, windows-1255 (Visual and Logical Hebrew)
- TIS-620 (Thai)
Requires Python 2.1 or later
However, some files will be valid in multiple encodings, so chardet is not a panacea.
Reliably? No.
In general, a byte sequence has no meaning unless you know how to interpret it -- this goes for text files, but also integers, floating point numbers, etc.
But, there are ways of guessing the encoding of a file, by looking at the byte order mark (if there is one) and the first chunk of the file (to see which encoding yields the most sensible characters). The chardet library is pretty good at this, but be aware it's only a heuristic, albeit a rather powerful one.
Try the chardet Python module, which is available on PyPI:
pip install chardet
Then run chardetect myfile.txt.
Chardet is based on the detection code used by Mozilla, so it should give reasonable results, provided that the input text is long enough for statistical analysis. Do read the project documentation.
As mentioned in comments it is quite slow, but some distributions also ship the original C++ version as @Xavier has found in https://superuser.com/a/609056. There is also a Java version somewhere.
I would use this simple command:
encoding=$(file -bi myfile.txt)
Or if you want just the actual character set (like utf-8):
encoding=$(file -b --mime-encoding myfile.txt)