What you're asking for is extremely hard. If possible, getting the user to specify the encoding is the best. Preventing an attack shouldn't be much easier or harder that way.
However, you could try doing this:
iconv(mb_detect_encoding($text, mb_detect_order(), true), "UTF-8", $text);
Setting it to strict might help you get a better result.
Answer from Jeff Day on Stack OverflowWhat you're asking for is extremely hard. If possible, getting the user to specify the encoding is the best. Preventing an attack shouldn't be much easier or harder that way.
However, you could try doing this:
iconv(mb_detect_encoding($text, mb_detect_order(), true), "UTF-8", $text);
Setting it to strict might help you get a better result.
In motherland Russia we have four popular encodings, so your question is in great demand here.
Only by character codes of symbols you can not detect the encoding, because code pages intersect. Some codepages in different languages have even full intersection. So, we need another approach.
The only way to work with unknown encodings is working with probabilities. So, we do not want to answer the question "what is encoding of this text?", we are trying to understand "what is most likely encoding of this text?".
One guy here in a popular Russian tech blog invented this approach:
Build the probability range of character codes in every encoding you want to support. You can build it using some big texts in your language (e.g., some fiction, use Shakespeare for English and Tolstoy for Russian, LOL). You will get something like this:
encoding_1:
190 => 0.095249209893009,
222 => 0.095249209893009,
...
encoding_2:
239 => 0.095249209893009,
207 => 0.095249209893009,
...
encoding_N:
charcode => probabilty
Next, you take text in an unknown encoding and for every encoding in your "probability dictionary" you search for the frequency of every symbol in the unknown-encoded text. Sum the probabilities of symbols. Encoding with the bigger rating is likely the winner. There are better results for bigger texts.
Btw, mb_detect_encoding certainly does not work. Yes, at all. Please, take a look of the mb_detect_encoding source code in "ext/mbstring/libmbfl/mbfl/mbfl_ident.c".
How to insert Unicode characters in MySQL using PHP?
What is UTF-8 character set?
What does UTF-8 stand for?
function convert_file_to_utf8($source, $target) {
$content=file_get_contents($source);
# detect original encoding
$original_encoding=mb_detect_encoding($content, "UTF-8, ISO-8859-1, ISO-8859-15", true);
# now convert
if ($original_encoding!='UTF-8') {
$content=mb_convert_encoding($content, 'UTF-8', $original_encoding);
}
$bom=chr(239) . chr(187) . chr(191); # use BOM to be on safe side
file_put_contents($target, $bom.$content);
}
before you can convert it to utf-8, you need to know what characterset it is. if you can't figure that out, you can't in any sane way convert it to utf8.. however, an insane way to convert it to utf-8, if the encoding cannot be determined, is to simply strip any bytes that doesn't happen to be valid in utf-8, you might be able to use that as a fallback...
warning, untested code (im suddenly in a hurry), but may look something like this:
foreach ( $datas as $data ) {
$encoding = guess_encoding ( $data );
if (empty ( $encoding )) {
// encoding cannot be determined...
// as a fallback, we simply strip any bytes that isnt valid utf-8...
// obviously this isn't a reliable conversion scheme.
// also this could probably be improved
$data = iconv ( "ASCII", "UTF-8//TRANSLIT//IGNORE", $text );
} else {
$data = mb_convert_encoding ( $data, 'UTF-8', $encoding );
}
$row [] = explode ( ',', $data );
}
function guess_encoding(string $str): string {
$blacklist = array (
'pass',
'auto',
'wchar',
'byte2be',
'byte2le',
'byte4be',
'byte4le',
'BASE64',
'UUENCODE',
'HTML-ENTITIES',
'7bit',
'8bit'
);
$encodings = array_flip ( mb_list_encodings () );
foreach ( $blacklist as $tmp ) {
unset ( $encodings [$tmp] );
}
$encodings = array_keys ( $encodings );
$detected = mb_detect_encoding ( $str, $encodings, true );
return ( string ) $detected;
}