Simply use the codecs module for writing the file:
import codecs
outputFile = codecs.open("textbase.tab", "w", "ISO-8859-1")
Of course, the strings you write have to be Unicode strings (type unicode), they won't be converted if they are plain str objects (which are basically just arrays of bytes). I guess you are reading the RTF file with the normal Python file object as well, so you might have to convert that to using codecs.open as well.
I have been using a small script to do some character replacement.
u = codecs.open(filename)
t = u.read().decode('iso-8859-1')
t = t.replace('º', 's') .
Still works on Ubuntu 12.04 flavor, python 3.2.
Now, if I try to run it on 14.04, python3.4, it quits at 't = u.read().decode('iso-8859-1')' with an error message:
File "/usr/lib/python3.4/codecs.py", line 319, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 55: invalid continuation byte
Any ideas what is different between 3.2 and 3.4?
Edited a few letters.
You shouldn't directly need the codecs module in Py3. Just pass the proper encoding to the regular open function.
with open(filename, encoding='iso-8859-1') as f:
t = f.read()
print(t.replace('º', 's'))
While reading, the content of the file is automatically decoded into a Unicode string. This is normal and desirable in Py3. Your t.replace('º', 's')) will do its thing correctly because t, 'º', and 's' are all Unicode strings. Life is so much easier in Py3 if you never muck around with bytes. So don't give any thought to encodings except when open-ing a file for reading or writing.
codecs.open will open the file using your system's default encoding. You're getting the error from the call to u.read(). You never even reach the call to decode right after (which should also raise an error because you can't decode strings).
Here's the change that caused the problem, and the associated issue. You're not really supposed to be able to use codecs.open without either giving an encoding or trying to open the file in binary mode. Your code only worked in 3.2 because of a bug.
character encoding - Python: How do I force iso-8859-1 file output? - Stack Overflow
python - How to read a "C source, ISO-8859 text" - Stack Overflow
Replace iso-8859-1 encoding symbols in python open module - Stack Overflow
Encoding characters with ISO 8859-1 in Python - Stack Overflow
Simply use the codecs module for writing the file:
import codecs
outputFile = codecs.open("textbase.tab", "w", "ISO-8859-1")
Of course, the strings you write have to be Unicode strings (type unicode), they won't be converted if they are plain str objects (which are basically just arrays of bytes). I guess you are reading the RTF file with the normal Python file object as well, so you might have to convert that to using codecs.open as well.
For me, io.open works a bit faster on python 2.7 for writes, and an order of magnitude faster for reads:
import io
with io.open("textbase.tab", "w", encoding="ISO-8859-1") as outputFile:
...
In python 3, you can just pass the encoding keyword arg to open.
With python 3.3 you can use the built in open function
open("myfile",encoding="ISO-8859-1")
You change the codec in the open() command; the ISO-8859 standard has multiple codecs, I picked Latin-1 for you here, but you may need to pick another one:
codecs.open('myfile', 'r', 'iso-8859-1').read()
See the codecs module for a list of valid codecs. Judging by the pastie data, iso-8859-1 is the correct codec to use, as it is suited for Scandinavian text.
Generally, without other sources, you cannot know what codec a file uses. At best, you can guess (which is what file does).
It's a mojibake case:
cmd
>NUL chcp 852
>dir_cp852.txt dir /C
type dir_cp852.txt | find /I "bytes free"
28 Dir(s) 832 467 206 144 bytes free
>NUL chcp 1252
type dir_cp852.txt | find /I "bytes free"
28 Dir(s) 832ÿ467ÿ206ÿ144 bytes free
Python
with open('dir_cp852.txt', 'r', encoding='iso-8859-1') as filename:
file_content = filename.read()
print(file_content[-52:])
28 Dir(s) 832ÿ467ÿ206ÿ144 bytes free
Solution:
with open('dir_cp852.txt', 'r', encoding='cp852') as filename:
file_content = filename.read()
print(file_content[-52:])
28 Dir(s) 832 467 206 144 bytes free
Note file_content[-52:] (in Python prompt):
' 28 Dir(s) 832\xa0467\xa0206\xa0144 bytes free\n'
shows character in mojibake: \xa0 (U+00A0, No-Break Space) with code 0xFF in Code page 852 (and more MS-DOS code pages).
Please note the /C switch in dir /C above (Display the thousand separator in file sizes).; I have overridden the default by (globally defined) set "DIRCMD=/-C".
The thousand separator in file sizes is defined in Control Panel\Clock and Region -> Region:reg query "HKCU\Control Panel\International" /v sThousand
Use ISO-8859-2 instead:
filename = open(f'/opt/PATH/{shorter}', 'r', encoding='iso-8859-2')
You can also try cp1250 or windows-1250. The Windows-1250 codepage is slightly different from ISO-8859-2.
ascii refers to the 7-bit US-ASCII codepage. That codepage wouldn't be able to open your file either.
If you used dir in a cmd shell, use chcp 65001 to switch to UTF8 before executing the script. Or use Powershell Core instead.
Codepages
As I explained in the comments, ÿ isn't some kind of encoding symbol. Single-byte codepages like Latin1 (aka ISO-8859-1), Central/Eastern European codepages like ISO-8859-2, Cyrillic etc simply encode characters to single byte values. Encoding symbols appear only in Unicode and markup languages like HTML and XML.
The character you posted ÿ is encoded to 255 (0xFF) in ISO-8859-1. In the old IBM DOS codepages 437 or 852 that byte corresponds to a non-breaking space. In the other ISO-8859- codepages including the Eastern European ISO-8859-2 the value is used for a dot.
What happened
I suspect the file was created by redirecting the dir output on Windows' cmd shell. dir in Powershell doesn't have this footer. dir in cmd will use the user's (yours) locale to format dates and numbers. This means you could get different results if someone used a custom format. Linux shells also allow such localization and customization.
The cmd shell is non-Unicode though, so when you redirected the output the shell used the current codepage, which matches the user's locale, to encode the byte values. To use UTF8 you have to explicitly change the shell codepage with
chcp 65001
Better alternatives
Windows Terminal and Powershell use Unicode by default like Windows itself and don't have such problems. Redirecting even allows you to specify encodings, handle results as objects or tables and even output the data as CSV or HTML. cmd is essentially a legacy shell.
In Powershell/Powershell Core you could use :
Dir | Export-CSV C:\Users\username\Desktop\FileList.csv
To export the directory list to a properly formatted CSV file in UTF8
When you're starting with a Unicode string, you need to encode rather than decode.
>>> def char_code(c):
return ord(c.encode('iso-8859-1'))
>>> print char_code(u'à')
224
For ISO-8859-1 in particular, you don't even need to encode it at all, since Unicode uses the ISO-8859-1 characters for its first 256 code points.
>>> print ord(u'à')
224
Edit: I see the problem now. You've given a source code encoding comment that indicates the source is in ISO-8859-1. However, I'll bet that your editor is actually working in UTF-8. The source code will be mis-interpreted, and the single-character string you think you created will actually be two characters. Try the following to see:
print len(u'à')
If your encoding is correct, it will return 1, but in your case it's probably 2.
You can get ord() for anything. As you might expect, ord(u'💩') works fine, provided you can represent the character properly in your source, and/or read it in a known encoding.
Your error message vaguely suggests that coding: iso-8859-1 is not actually true, and the file's encoding is actually something else (UTF-8 or UTF-16 would be my guess).
The canonical must-read on character encoding in Python is http://nedbatchelder.com/text/unipain.html