[ TLDR => Application encoding a confusing issue, but this document from Oracle should help. ]
First a few important general points about specifying the encoding by setting the System Property file.encoding at run time:
It's use is not formally supported, and never has been. From a Java Bug Report in 1998:
The "file.encoding" property is not required by the J2SE platform specification; it's an internal detail of Sun's implementations and should not be examined or modified by user code. It's also intended to be read-only; it's technically impossible to support the setting of this property to arbitrary values on the command line or at any other time during program execution.
There is a draft JEP (JDK Enhancement Proposal), JDK-8187041 Use UTF-8 as default Charset, which proposes:
Use UTF-8 as the Java virtual machine's default charset so that APIs that depend on the default charset behave consistently across all platforms.
It doesn't necessarily make sense to claim that "This application uses encoding {x}" since there may be multiple encodings associated with an application, which can be addressed in different ways, including:
- The file encoding for console output.
- The file encoding of the application's source files.
- The file encoding(s) for file I/O.
- The file encoding of file paths.
All that said, Oracle specify all encodings supported by Java SE 8. I can't find a corresponding document for more recent JDK versions. Note that:
- Encodings can be environment specific, based on locale, operating system, Java version, etc.
- Almost every encoding has at least one alias. For example, the encoding name for simplified Chinese is GBK, but you could also use CP936 or windows-936.
- Most encoding are non Unicode since Unicode encoding names contain the string "UTF".
- An encoding name can vary depending on how the application is processing files (
java.nioAPIs vs.java.io/java.langAPIs.). For example, if performing some I/O on Turkish files on Windows:- If the
java.nio.*classes are used, specify -Dfile.encoding=windows-1254 at runtime. - If the
java.lang.*&java.io.*classes are used, specify -Dfile.encoding=Cp1254 at runtime.
- If the
This DZone article provides a useful piece of code to show how setting -Dfile.encoding at runtime can impact various settings:
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.Charset;
import java.util.Locale;
import static java.lang.System.out;
/**
* Demonstrate default Charset-related details.
*/
public class CharsetDemo
{
/**
* Supplies the default encoding without using Charset.defaultCharset()
* and without accessing System.getProperty("file.encoding").
*
* @return Default encoding (default charset).
*/
public static String getEncoding()
{
final byte [] bytes = {'D'};
final InputStream inputStream = new ByteArrayInputStream(bytes);
final InputStreamReader reader = new InputStreamReader(inputStream);
final String encoding = reader.getEncoding();
return encoding;
}
public static void main(final String[] arguments)
{
out.println("Default Locale: " + Locale.getDefault());
out.println("Default Charset: " + Charset.defaultCharset());
out.println("file.encoding; " + System.getProperty("file.encoding"));
out.println("sun.jnu.encoding: " + System.getProperty("sun.jnu.encoding"));
out.println("Default Encoding: " + getEncoding());
}
}
Here's some sample output when specifying -Dfile.encoding=860 (an alias for MS-DOS Portuguese) using Java 12 on Windows 10:
run:
Default Locale: en_US
Default Charset: IBM860
file.encoding: 860
sun.jnu.encoding: Cp1252
Default Encoding: Cp860
BUILD SUCCESSFUL (total time: 0 seconds)
Test the encoding you plan to specify at run time on all target platforms. You may get unexpected results. For example, when I run the code above on Windows 10 with -Dfile.encoding=IBM864 (PC Arabic) it works, but fails with -Dfile.encoding=IBM420 (IBM Arabic).
Answer from skomisa on Stack OverflowNeed Example of Unicode and Non-unicode data – SQLServerCentral Forums
java - Find all non unicode characters in file - Stack Overflow
Unicode Noncharacters - Stack Overflow
How do you write a character not present in unicode?
The official source can already be found in http://unicode.org/charts/index.html; search up for "Noncharacters in Charts." In fact, the noncharacters at the end of Plane 3 to D [as of Unicode 12.1] are the only designated code points in these planes.
There are exactly 66 noncharacters in Unicode. There are 34 noncharacters residing at the final two code points of each of the 17 planes, and there is an additional contiguous range of 32 noncharacters from U+FDD0 to U+FDEF in the Arabic Presentation Forms-B block.
Any code point ending with FFFE or FFFF is a noncharacter. For the exceptions, any 4-digit code point beginning with FDD or FDE is a noncharacter.
I'll enumerate the noncharacters:
- FDD0-FDEF [These 32 are designated in Unicode 3.1, to allocate more code points for internal use]
- FFFE [Probably the most notable one, this one is involved in BOM usage]
- FFFF [Can be used as a sentinel, equal to -1 in a 16-bit signed int]
- XFFFE [16 of them in the supplementary planes; X is a hexadecimal digit or 10]
- XFFFF [16 of them in the supplementary planes; X is a hexadecimal digit or 10]
..., as I can't tell where each plane ends.
Every plane by definition ends at U+xxFFFF.
On Unicodes website it refers to the last two characters of every plane being non characters.
No. The Unicode Standard Version 9.0 - Core Specification says (in section 23.7 Noncharacters):
The Unicode Standard sets aside 66 noncharacter code points. The last two code points of each plane are noncharacters: U+FFFE and U+FFFF on the BMP, U+1FFFE and U+1FFFF on Plane 1, and so on, up to U+10FFFE and U+10FFFF on Plane 16, for a total of 34 code points. In addition, there is a contiguous range of another 32 noncharacter code points in the BMP: U+FDD0..U+FDEF. For historical reasons, the range U+FDD0..U+FDEF is contained within the Arabic Presentation Forms-A block, but those noncharacters are not “Arabic noncharacters” or “right-to-left noncharacters,” and are not distinguished in any other way from the other noncharacters, except in their code point values.
Note the keyword "code points", not "characters", they are always U+xxFFFE and U+xxFFFF.
Hello, I'm working on research about Jersey Dutch. It uses a lot of diacritics and symbols for its letters which are important for proununciation. I found out that a 'g' +'´' doesn't work in unicode, it should be "". How would you write said character?