Why don't you read the file and write it as UTF-8? You can do that in Python.
#to support encodings
import codecs
#read input file
with codecs.open(path, 'r', encoding = 'utf8') as file:
lines = file.read()
#write output file
with codecs.open(path, 'w', encoding = 'utf8') as file:
file.write(lines)
Answer from 3Ducker on Stack OverflowWhy don't you read the file and write it as UTF-8? You can do that in Python.
#to support encodings
import codecs
#read input file
with codecs.open(path, 'r', encoding = 'utf8') as file:
lines = file.read()
#write output file
with codecs.open(path, 'w', encoding = 'utf8') as file:
file.write(lines)
I appreciate that this is an old question but having just resolved a similar problem recently I thought I would share my solution.
I had a file being prepared by one program that I needed to import in to an sqlite3 database but the text file was always 'ANSI' and sqlite3 requires UTF-8.
The ANSI encoding is recognised as 'mbcs' in python and therefore the code I have used, ripping off something else I found is:
blockSize = 1048576
with codecs.open("your ANSI source file.txt","r",encoding="mbcs") as sourceFile:
with codecs.open("Your UTF-8 output file.txt","w",encoding="UTF-8") as targetFile:
while True:
contents = sourceFile.read(blockSize)
if not contents:
break
targetFile.write(contents)
The below link contains some information on the encoding types that I found on my research
https://docs.python.org/2.4/lib/standard-encodings.html
encoding - convert ansi escape to utf-8 in python - Stack Overflow
Python 3 ANSI to UTF-8
UTF-8 and ANSI encoding issue
python - From ansi encoding to utf8 (and hex bytes) - Stack Overflow
Try this
#read input file
with codecs.open('USERS.CSV', 'r', encoding = 'latin-1') as file:
lines = file.read()
#write output file
with codecs.open('1_UserPython.CSV', 'w', encoding = 'utf_8_sig') as file:
file.write(lines)
To convert a file from utf8 to cp1252:
import io
with io.open(src_path, mode="r", encoding="utf8") as fd:
content = fd.read()
with io.open(dst_path, mode="w", encoding="cp1252") as fd:
fd.write(content)
» pip install ansipants
You can try this code:
import codecs
import os
import sys
filePathSrc="C:\\222\\3" # Path to the folder with files to convert
for root, dirs, files in os.walk(unicode(filePathSrc)):
for fn in files:
if fn[-4:] == '.srt': # Specify type of the files
filename = unicode(root + "\\" + fn)
with codecs.open(filename,'r', encoding = "Windows-1251") as f:
text = f.read()
# process Unicode text
with codecs.open(filename,'w',encoding='utf8') as f:
# f.write(u'\uFEFF') # BOM mark optional
f.write(text)
Points:
import codecsadded to work with files in Pythonos.walk(unicode(filePathSrc))is given a Unicode path to return Unicode file names- You should specify the correct encoding for your files instead of
Windows-1251in thewith codecs.open(filename,'r', encoding = "Windows-1251")code. - If the folder
filePathSrcvariable should have Unicode chars, convert them to\uXXXXnotation (you can do that easily with r12a Unicode Converter from the JavaScript escapes field). Say, your folder name is7 Minutes 2014{جنایی}{7 دقیقه}. You paste it to the green field, and click Convert. Then, grab the string from the JavaScript escapes field and use it forfilePathSrcvariable while also pre-pending the string withu""prefix. It will look asfilePathSrc=u"c:\\222\\7 Minutes 2014{\u062C\u0646\u0627\u06CC\u06CC}{7 \u062F\u0642\u06CC\u0642\u0647}". Then, instead ofos.walk(unicode(filePathSrc))useos.walk(filePathSrc)since the string we pass is already Unicode.
If you use a Unicode path in os.walk() it will return Unicode paths and filenames. Notepad isn't required to convert the files. Below is code that will work in Python 2 and Python 3 since it wasn't specified.
Note that strings are Unicode by default in Python 3, but the from __future__ makes Python 2 strings default Unicode where normally they are byte strings. Making sure you use Unicode strings everywhere is important.
io.open is the Python 3 version of open, but is available in Python 2 as well. It opens files with "ANSI" encoding by default. locale.getpreferredencoding() can be used to determine the exact encoding. It is cp1252 on US Windows. read() will return the file data decoded into Unicode.
The encoding utf-8-sig will prepend a UTF-8-encoded BOM character (which Windows tends to like) and encode the written data using UTF-8. If the BOM is not desired, use utf8 instead.
from __future__ import unicode_literals
import os
import io
import fnmatch
filePathSrc = r'C:\test'
for root, dirs, files in os.walk(filePathSrc):
for fn in fnmatch.filter(files,'*.srt'):
fullname = os.path.join(root,fn)
with io.open(fullname) as f:
data = f.read()
with io.open(fullname,'w',encoding='utf-8-sig') as f:
f.write(data)
Go only has UTF-8 strings. You can convert something to a UTF8 string using the conversion described here from a byte[]:
http://golang.org/doc/go_spec.html#Conversions
Here is newer method.
package main
import (
"bytes"
"fmt"
"io/ioutil"
"golang.org/x/text/encoding/traditionalchinese"
"golang.org/x/text/transform"
)
func Decode(s []byte) ([]byte, error) {
I := bytes.NewReader(s)
O := transform.NewReader(I, traditionalchinese.Big5.NewDecoder())
d, e := ioutil.ReadAll(O)
if e != nil {
return nil, e
}
return d, nil
}
func main() {
s := []byte{0xB0, 0xAA}
b, err := Decode(s)
fmt.Println(string(b))
fmt.Println(err)
}
I were use iconv-go to do such convert, you must know what's your ANSI code page, in my case, it is 'big5'.
package main
import (
"fmt"
//iconv "github.com/djimenez/iconv-go"
iconv "github.com/andelf/iconv-go"
"log"
)
func main() {
ibuf := []byte{170,76,80,67}
var obuf [256]byte
// Method 1: use Convert directly
nR, nW, err := iconv.Convert(ibuf, obuf[:], "big5", "utf-8")
if err != nil {
log.Fatalln(err)
}
log.Println(nR, ibuf)
log.Println(obuf[:nW])
fmt.Println(string(obuf[:nW]))
// Method 2: build a converter at first
cv, err := iconv.NewConverter("big5", "utf-8")
if err != nil {
log.Fatalln(err)
}
nR, nW, err = cv.Convert(ibuf, obuf[:])
if err != nil {
log.Fatalln(err)
}
log.Println(string(obuf[:nW]))
}