java - Valid/invalid non-ascii and invalid ascii characters - Stack Overflow
regex - How do I grep for all non-ASCII characters? - Stack Overflow
ide - not used characters from ANSI characters set - Stack Overflow
How to fix Non ANSI Characters | Trainz
There are 128 valid basic ASCII characters, mapped to the values 0 (the NUL byte) to 127 (the DEL character). See here.
The word 'character' must be used wisely. The definition of 'character' is a special one. For example, the è, is that one character? Or is it two characters (e and `)? It depends.
Secondly, a sequence of characters is completely independent from its encoding. For simplicity, I assume that each byte is interpreted as one character.
You can determine if a byte can be parsed as an ASCII character, you can simply do this:
byte[] bytes = "Bj��rk����oacute�".getBytes();
for (byte b : bytes) {
// What's happening here? A byte that is in the range from 0 to 127 is
// valid, and other values are invalid. A byte in Java is signed, that
// means that valid ranges are from -128 to 127.
if (b >= 0) {
System.out.println("Valid ASCII");
}
else {
System.out.println("Invalid ASCII");
}
}
Some background
As Java was invented, a very important design decision was that text in java would be Unicode: a numbering system of all graphemes in the world. Hence char is two bytes (in UTF-16, one of the Unicode "universal character set transformation format"). And byte is a distinct type for binary data.
Unicode numbers all symbols, so-called code points, like ♫, as U+266B. Those numbers reaching the three byte integers. Hence code points in java are represented as int.
ASCII is a 7-bits subset of Unicode UTF-8, 0 - 127.
UTF-8 is a multibyte Unicode format, where ASCII is a valid subset, and higher symbols
Validity
You were asked to identify "invalid" characters = wrongly produced code points. You could also identify code parts that produce invalid characters. (Easier.)
In the above � is a place holder character (like ?) that substitutes a code point not being representable in the current character set. If the code produced a ? as place holder, one cannot guess whether substitution took place. For some west European languages the encoding is Windows-1252 (Cp1252, MS Windows Latin-1) having. You can check whether a code point from a String can be converted to that Charset.
Then remain false positives, wrong characters that however exist in Cp1252. That could be a multi-byte code sequence of UTF-8, interpreted as several Window-1252 characters. So: an acceptable non-ASCII char adjacent to a unacceptable non-ASCII char is suspect too. That means you need to list the special characters in your language, and extras: like special quotes, in English borrows like ç, ñ.
For MS-Windows Latin-1 (an altered ISO Latin-1) something like:
boolean isSuspect(char ch) {
if (ch < 32) {
return "\f\n\r\t".indexOf(ch) != -1;
} else if (ch >= 127) {
return false;
} else {
return suspects.get((int) ch); // Better use a positive list.
}
}
static BitSet suspects = new BitSet(256);
static {
...
}
You can use the command:
LC_ALL=C grep --color='auto' -P -n "[\x80-\xFF]" file.xml
This will give you the line number, and will highlight non-ascii chars in red.
In some systems, depending on your settings, the above will not work, so you can grep by the inverse
LC_ALL=C grep --color='auto' -P -n "[^\x00-\x7F]" file.xml
Note also, that the important bit is the -P flag which equates to --perl-regexp: so it will interpret your pattern as a Perl regular expression. It also says that
this is highly experimental and grep -P may warn of unimplemented features.
Instead of making assumptions about the byte range of non-ASCII characters, as most of the above solutions do, it's slightly better IMO to be explicit about the actual byte range of ASCII characters instead.
So the first solution for instance would become:
grep --color='auto' -P -n '[^\x00-\x7F]' file.xml
(which basically greps for any character outside of the hexadecimal ASCII range: from \x00 up to \x7F)
On Mountain Lion that won't work (due to the lack of PCRE support in BSD grep), but with pcre installed via Homebrew, the following will work just as well:
pcregrep --color='auto' -n '[^\x00-\x7F]' file.xml
Any pros or cons that anyone can think off?
You seem to be talking about Windows-1252, which is just one of many "ANSI" code pages Windows can use, and it's probably not used outside of Windows. Don't tie a new product to an obsolete technology. Not supporting Unicode (be it UTF-16le or UTF-8) is unacceptable for a programming language.
While it's rather moot to answer the direct question, the answer is no, you cannot assume they will be treated as whitespace. Some may. Some may replace with a space. Some may replace with another glyph. Some may use special colours. Some may give a warning. Some may not load the file.
By the way, if you are referring to Windows-1252, only 0x81, 0x8D, 0x8F, 0x90, 0x9D aren't assigned.
You shouldn't assume any specific behavior, as it will depend on the display widget and quite possibly on the font. Either preprocess the text for display or use an out-of-band markup mechanism (for example, many text field widgets let you attach attributes to runs of text).
I'm trying to find non-ASCII characters in a buffer. I have visited the buffer literally and then try C-M-s and type
[:nonascii:]but it finds lots of ASCII characters. I candescribe-charon what it finds and the code points are still ASCII. I have also tried M-S-:(re-search-forward "[:nonascii:]")and it also finds ascii characters. I have tried buffers literally and regular and both do it. Does anyone know what I might be doing wrong? The above methods detect the "c", "a"s, and "I" as non-ASCII in the following string "01165cam a2200349Ia". It does this in the buffers for files I'm working with and in any Emacs generated buffer such as the describe-char results. Thanks.