[ TLDR => Application encoding a confusing issue, but this document from Oracle should help. ]

First a few important general points about specifying the encoding by setting the System Property file.encoding at run time:

  • It's use is not formally supported, and never has been. From a Java Bug Report in 1998:

    The "file.encoding" property is not required by the J2SE platform specification; it's an internal detail of Sun's implementations and should not be examined or modified by user code. It's also intended to be read-only; it's technically impossible to support the setting of this property to arbitrary values on the command line or at any other time during program execution.

  • There is a draft JEP (JDK Enhancement Proposal), JDK-8187041 Use UTF-8 as default Charset, which proposes:

    Use UTF-8 as the Java virtual machine's default charset so that APIs that depend on the default charset behave consistently across all platforms.

  • It doesn't necessarily make sense to claim that "This application uses encoding {x}" since there may be multiple encodings associated with an application, which can be addressed in different ways, including:

    • The file encoding for console output.
    • The file encoding of the application's source files.
    • The file encoding(s) for file I/O.
    • The file encoding of file paths.

All that said, Oracle specify all encodings supported by Java SE 8. I can't find a corresponding document for more recent JDK versions. Note that:

  • Encodings can be environment specific, based on locale, operating system, Java version, etc.
  • Almost every encoding has at least one alias. For example, the encoding name for simplified Chinese is GBK, but you could also use CP936 or windows-936.
  • Most encoding are non Unicode since Unicode encoding names contain the string "UTF".
  • An encoding name can vary depending on how the application is processing files (java.nio APIs vs. java.io/java.lang APIs.). For example, if performing some I/O on Turkish files on Windows:
    • If the java.nio.* classes are used, specify -Dfile.encoding=windows-1254 at runtime.
    • If the java.lang.* & java.io.* classes are used, specify -Dfile.encoding=Cp1254 at runtime.

This DZone article provides a useful piece of code to show how setting -Dfile.encoding at runtime can impact various settings:

import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.Charset;
import java.util.Locale;
import static java.lang.System.out;
/**
 * Demonstrate default Charset-related details.
 */
public class CharsetDemo
{
   /**
    * Supplies the default encoding without using Charset.defaultCharset()
    * and without accessing System.getProperty("file.encoding").
    *
    * @return Default encoding (default charset).
    */
   public static String getEncoding()
   {
      final byte [] bytes = {'D'};
      final InputStream inputStream = new ByteArrayInputStream(bytes);
      final InputStreamReader reader = new InputStreamReader(inputStream);
      final String encoding = reader.getEncoding();
      return encoding;
   }
   public static void main(final String[] arguments)
   {
      out.println("Default Locale:   " + Locale.getDefault());
      out.println("Default Charset:  " + Charset.defaultCharset());
      out.println("file.encoding;    " + System.getProperty("file.encoding"));
      out.println("sun.jnu.encoding: " + System.getProperty("sun.jnu.encoding"));
      out.println("Default Encoding: " + getEncoding());
   }
}

Here's some sample output when specifying -Dfile.encoding=860 (an alias for MS-DOS Portuguese) using Java 12 on Windows 10:

run:
Default Locale:   en_US
Default Charset:  IBM860
file.encoding:    860
sun.jnu.encoding: Cp1252
Default Encoding: Cp860
BUILD SUCCESSFUL (total time: 0 seconds)

Test the encoding you plan to specify at run time on all target platforms. You may get unexpected results. For example, when I run the code above on Windows 10 with -Dfile.encoding=IBM864 (PC Arabic) it works, but fails with -Dfile.encoding=IBM420 (IBM Arabic).

Answer from skomisa on Stack Overflow
🌐
FileFormat
fileformat.info › info › unicode › font › symbol › nonunicode.htm
Non-unicode characters supported by the Symbol font
Non-unicode characters supported by the Symbol font · Count: 188 · Terms of Service | Privacy Policy | Contact Info
Top answer
1 of 1
1

[ TLDR => Application encoding a confusing issue, but this document from Oracle should help. ]

First a few important general points about specifying the encoding by setting the System Property file.encoding at run time:

  • It's use is not formally supported, and never has been. From a Java Bug Report in 1998:

    The "file.encoding" property is not required by the J2SE platform specification; it's an internal detail of Sun's implementations and should not be examined or modified by user code. It's also intended to be read-only; it's technically impossible to support the setting of this property to arbitrary values on the command line or at any other time during program execution.

  • There is a draft JEP (JDK Enhancement Proposal), JDK-8187041 Use UTF-8 as default Charset, which proposes:

    Use UTF-8 as the Java virtual machine's default charset so that APIs that depend on the default charset behave consistently across all platforms.

  • It doesn't necessarily make sense to claim that "This application uses encoding {x}" since there may be multiple encodings associated with an application, which can be addressed in different ways, including:

    • The file encoding for console output.
    • The file encoding of the application's source files.
    • The file encoding(s) for file I/O.
    • The file encoding of file paths.

All that said, Oracle specify all encodings supported by Java SE 8. I can't find a corresponding document for more recent JDK versions. Note that:

  • Encodings can be environment specific, based on locale, operating system, Java version, etc.
  • Almost every encoding has at least one alias. For example, the encoding name for simplified Chinese is GBK, but you could also use CP936 or windows-936.
  • Most encoding are non Unicode since Unicode encoding names contain the string "UTF".
  • An encoding name can vary depending on how the application is processing files (java.nio APIs vs. java.io/java.lang APIs.). For example, if performing some I/O on Turkish files on Windows:
    • If the java.nio.* classes are used, specify -Dfile.encoding=windows-1254 at runtime.
    • If the java.lang.* & java.io.* classes are used, specify -Dfile.encoding=Cp1254 at runtime.

This DZone article provides a useful piece of code to show how setting -Dfile.encoding at runtime can impact various settings:

import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.Charset;
import java.util.Locale;
import static java.lang.System.out;
/**
 * Demonstrate default Charset-related details.
 */
public class CharsetDemo
{
   /**
    * Supplies the default encoding without using Charset.defaultCharset()
    * and without accessing System.getProperty("file.encoding").
    *
    * @return Default encoding (default charset).
    */
   public static String getEncoding()
   {
      final byte [] bytes = {'D'};
      final InputStream inputStream = new ByteArrayInputStream(bytes);
      final InputStreamReader reader = new InputStreamReader(inputStream);
      final String encoding = reader.getEncoding();
      return encoding;
   }
   public static void main(final String[] arguments)
   {
      out.println("Default Locale:   " + Locale.getDefault());
      out.println("Default Charset:  " + Charset.defaultCharset());
      out.println("file.encoding;    " + System.getProperty("file.encoding"));
      out.println("sun.jnu.encoding: " + System.getProperty("sun.jnu.encoding"));
      out.println("Default Encoding: " + getEncoding());
   }
}

Here's some sample output when specifying -Dfile.encoding=860 (an alias for MS-DOS Portuguese) using Java 12 on Windows 10:

run:
Default Locale:   en_US
Default Charset:  IBM860
file.encoding:    860
sun.jnu.encoding: Cp1252
Default Encoding: Cp860
BUILD SUCCESSFUL (total time: 0 seconds)

Test the encoding you plan to specify at run time on all target platforms. You may get unexpected results. For example, when I run the code above on Windows 10 with -Dfile.encoding=IBM864 (PC Arabic) it works, but fails with -Dfile.encoding=IBM420 (IBM Arabic).

Discussions

java - Find all non unicode characters in file - Stack Overflow
So i have a textfile that includes non unicode characters. For example "pr�s-*" ESt Präs How do I print them out, but only them. I know this java method for replacing it String resultString = More on stackoverflow.com
🌐 stackoverflow.com
May 24, 2017
Unicode Noncharacters - Stack Overflow
Obviously 0xFFFE and 0xFFFF is a non character, as well as 0x10FFFE and 0x10FFFF, but I can't find a complete list as to where the last characters are of each plane, as I can't tell where each plane ends. On Unicodes website it refers to the last two characters of every plane being non characters. More on stackoverflow.com
🌐 stackoverflow.com
How do you write a character not present in unicode?
Is this the character you're looking for? ǵ LATIN SMALL LETTER G WITH ACUTE LOWERCASE LETTER LEFT-TO-RIGHT 501 U+01F5 More on reddit.com
🌐 r/linguistics
23
34
May 18, 2023
What is the most useless non-deprecated Unicode character?
U+1F574 MAN IN BUSINESS SUIT LEVITATING obviously. More seriously, it's hard to say what people will do with a Unicode character. For example, U+A66E CYRILLIC LETTER MULTIOCULAR O was introduced to encode a character that occurs in a single phrase, in a single text, in a single manuscript written in Church Slavonic (a copy of the Book of Psalms) in 1429 (also, most fonts get it wrong, it needs to have 10 eyes, most have only 7 because of an incorrect initial submission). But somehow various people found it useful to decorate their name or various things of the sort. There's also U+237C RIGHT ANGLE WITH DOWNWARDS ZIGZAG ARROW , for example, which is one of the many characters which were included into Unicode because it was in some prior standard, but nobody really knows what purpose it's supposed to serve. More on reddit.com
🌐 r/Unicode
19
20
June 8, 2024
🌐
SQLServerCentral
sqlservercentral.com › home › topics › need example of unicode and non-unicode data
Need Example of Unicode and Non-unicode data – SQLServerCentral Forums
March 24, 2008 - So if I'm designing a table thats going to get very large (supporting english only) then one design consideration should be stay with non-unicode data types? ... That is correct. Each Unicode character takes 2 bytes of storage (2 8-bit words) and each non-Unicode character takes 1 byte (1 8-bit word) of storage.
🌐
Oracle
docs.oracle.com › cd › F26413_47 › books › GlobDep › c-NonUnicode-Traditional-Character-Sets-xu1054274.html
Non-Unicode (Traditional) Character Sets
Before the emergence of Unicode, ... Examples of non-Unicode character sets are Code Page 1252 for languages spoken in Western European countries as well as in the Americas and elsewhere, and Code Page 932 for the Japanese language....
🌐
Oracle
docs.oracle.com › cd › B40099_02 › books › GlobDep › GlobDepOverview10.html
Bookshelf v8.0: Non-Unicode (Traditional) Character Sets
Siebel Global Deployment Guide > Overview of Global Deployments > About Supported Character Sets > · This topic is part of About Supported Character Sets
🌐
Stack Overflow
stackoverflow.com › questions › 38912331 › find-all-non-unicode-characters-in-file
java - Find all non unicode characters in file - Stack Overflow
May 24, 2017 - If you work with a text copied from such an editor, you may want to find only the repl. character, which has the Unicode code point of U+FFFD. If you want to work with the original text, where you get some of the bytes displayed as the repl. char, than of course it's more complicated - you probably should at 1st find out what the correct encoding is (where those bytes really mean something). ... You may use a Matcher#find to find and print all these non-ASCII chars with your [^\\x00-\\x7F] regex or \\P{ASCII}:
🌐
Irfan's World
irfansworld.wordpress.com › 2011 › 01 › 25 › what-is-unicode-and-non-unicode-data-formats
What is Unicode and non Unicode data formats? | Irfan's World
January 31, 2011 - In my earlier post, I have explained about data. In this post, I am going to explain you some basics about Unicode and non Unicode formats: Unicode : A Unicode character takes more bytes to store t…
Find elsewhere
🌐
IIT Kanpur
iitk.ac.in › esc101 › 05Aug › tutorial › i18n › text › convertintro.html
Converting Non-Unicode Text
To indicate Unicode characters that cannot be represented in ASCII, such as ö, we used the \uXXXX escape sequence. Each X in the escape sequence is a hexadecimal digit. The following example shows how to indicate the ö character with an escape sequence:
🌐
Oracle
docs.oracle.com › cd › E88140_01 › books › GlobDep › GlobDepOverview11.html
Siebel Innovation Pack 2017: Non-Unicode (Traditional) Character Sets
Siebel Global Deployment Guide > Overview of Global Deployments > About Supported Character Sets > · This topic is part of About Supported Character Sets
Top answer
1 of 4
4

The official source can already be found in http://unicode.org/charts/index.html; search up for "Noncharacters in Charts." In fact, the noncharacters at the end of Plane 3 to D [as of Unicode 12.1] are the only designated code points in these planes.

There are exactly 66 noncharacters in Unicode. There are 34 noncharacters residing at the final two code points of each of the 17 planes, and there is an additional contiguous range of 32 noncharacters from U+FDD0 to U+FDEF in the Arabic Presentation Forms-B block.

Any code point ending with FFFE or FFFF is a noncharacter. For the exceptions, any 4-digit code point beginning with FDD or FDE is a noncharacter.

I'll enumerate the noncharacters:

  • FDD0-FDEF [These 32 are designated in Unicode 3.1, to allocate more code points for internal use]
  • FFFE [Probably the most notable one, this one is involved in BOM usage]
  • FFFF [Can be used as a sentinel, equal to -1 in a 16-bit signed int]
  • X⁠FFFE [16 of them in the supplementary planes; X is a hexadecimal digit or 10]
  • X⁠FFFF [16 of them in the supplementary planes; X is a hexadecimal digit or 10]
2 of 4
2

..., as I can't tell where each plane ends.

Every plane by definition ends at U+xxFFFF.

On Unicodes website it refers to the last two characters of every plane being non characters.

No. The Unicode Standard Version 9.0 - Core Specification says (in section 23.7 Noncharacters):

The Unicode Standard sets aside 66 noncharacter code points. The last two code points of each plane are noncharacters: U+FFFE and U+FFFF on the BMP, U+1FFFE and U+1FFFF on Plane 1, and so on, up to U+10FFFE and U+10FFFF on Plane 16, for a total of 34 code points. In addition, there is a contiguous range of another 32 noncharacter code points in the BMP: U+FDD0..U+FDEF. For historical reasons, the range U+FDD0..U+FDEF is contained within the Arabic Presentation Forms-A block, but those noncharacters are not “Arabic noncharacters” or “right-to-left noncharacters,” and are not distinguished in any other way from the other noncharacters, except in their code point values.

Note the keyword "code points", not "characters", they are always U+xxFFFE and U+xxFFFF.

🌐
SEOZoom
seozoom.com › home › blog › non-ascii characters and special characters: what they are and how to use them
Non-ASCII and special characters: guidance and tips for the site
May 23, 2024 - For example, a text file saved in UTF-8 that contains emoji, when opened with an encoding that does not support emoji, might show: ... If a database is not configured to support UTF-8 (or another Unicode encoding) and you try to insert non-ASCII characters, they may be saved incorrectly.
🌐
Oracle
docs.oracle.com › cd › E14004_01 › books › GlobDep › GlobDepOverview11.html
Bookshelf v8.1/8.2: Non-Unicode (Traditional) Character Sets
Siebel Global Deployment Guide > Overview of Global Deployments > About Supported Character Sets > · This topic is part of About Supported Character Sets
🌐
Wikipedia
en.wikipedia.org › wiki › Specials_(Unicode_block)
Specials (Unicode block) - Wikipedia
June 6, 2026 - U+FFFC  OBJECT REPLACEMENT ... for example in a compound document. U+FFFD � REPLACEMENT CHARACTER used to replace an unknown, unrecognised, or unrepresentable character · U+FFFE <noncharacter-FFFE> not a character. U+FFFF <noncharacter-FFFF> not a character. ... U+FFFF <noncharacter-FFFF> are noncharacters, meaning they are reserved but do not cause ill-formed Unicode ...
🌐
Oracle
docs.oracle.com › javase › tutorial › i18n › text › convertintro.html
Converting Non-Unicode Text (The Java™ Tutorials > Internationalization > Working with Text)
To indicate Unicode characters that cannot be represented in ASCII, such as ö, we used the \uXXXX escape sequence. Each X in the escape sequence is a hexadecimal digit. The following example shows how to indicate the ö character with an escape sequence:
🌐
Quora
quora.com › What-is-a-non-UTF-8-character
What is a non- UTF-8 character? - Quora
Answer (1 of 2): Thomas Phinney's answer is not correct, but its incorrectness hinges on definition. https://stackoverflow.com/questions/58210104/is-there-such-a-thing-as-non-utf8-character?answertab=trending#tab-top UTF-8 is a scheme for encoding any Unicode character (represented by a number b...
🌐
Blogger
oraclepeoplesoftadmins.blogspot.com › 2017 › 02 › unicode-and-non-unicode.html
Oracle Fusion Technical (HCM, FSCM) and Peoplesoft Administrator: Unicode And Non-Unicode
February 24, 2017 - Using non-Unicode it is easy to store languages like ‘English’ but not other Asian languages that need more bits to store correctly otherwise truncation will occur.
🌐
Wikipedia
en.wikipedia.org › wiki › List_of_Unicode_characters
List of Unicode characters - Wikipedia
4 days ago - As it is not technically possible ... characters in the Multilingual European Character Set 2 (MES-2) subset, and some additional related characters. (The term Unicode character was coined to categorise characters that do not also have ASCII code points.)...