project management - Are there historical problems with non-ASCII identifier characters in code? - Software Engineering Stack Exchange
injection - Is it a good idea to use non-ASCII names in the U.S.? - Information Security Stack Exchange
[deleted by user]
Searching for nonascii characters
What is the difference between ASCII and non-ASCII?
What counts as a non-ASCII character?
What is a non-ASCII character finder?
There are historical reasons for this guidance, that are mainly related to the lack of uniform encoding standard.
Encoding issues
Unicode dates back to the 90s'. Before it became mainstream, there was no global standard way to encode λ, é, or ä. The western character sets used to be limited to a single byte (octet to be more precise). This leaves room for only 128 characters on top of the ASCII ones. But there are many more accented local characters and greek letters than the 128 that were available.
This is why the ISO-8859 character sets were standardized in several variants. People used country-dependent settings. Suppose a German developper worked in ISO 8859-1 and had an ß in the identifier. This was encoded as 0xDF in the source file. If a greek developer opened the same source, using an ISO 8859-7 setting, the identifier would be displayed with an ί instead of ß. If the greek developer would type a ß it would lead to 0xE2 in the source file, which a French developer would then read as â. This was a total mess guaranteed. And when an US colleague looked at it, it just showed a dot, or a question mark, or, a semi-graphic character (after 1981, when extended ASCII became popular).
Tool support
The lack of uniformity triggered practical annoyances in text editors. I remember for example that the word-by-word move viewed non-ascii as a word separator. So the navigation was not as smooth as with ascii identifiers.
More seriously, there was a lacking tool support. I remember the first linker on MS-DOS had constraints about the length of the symbol identifiers and it was limited to ASCII character sets. This is btw not so old story according Wikipedia about GNU Compiler collection (see also this SO question):
Although the C++ language requires support for non-ASCII Unicode characters in identifiers, the feature has only been supported since GCC 10. As with the existing handling of string literals, the source file is assumed to be encoded in UTF-8. The feature is optional in C, but has been made available too since this change.
Momentum, prejudice and internationalization
To prevent all these nasty issues, a lot of coding standards pragmatically recommended the use of ASCII characters for identifiers. This creates some momentum.
Moreover there is a broad consensus for using English language in identifiers in the context of international projects, or open source projects looking for a broad community. Having this kind of expectations create some prejudice against opening up to unicode characters.
But not all projects have an international audience. There are lots of teams out there working in a local context and using native language in comments, in git commits and even in identifiers. A study of some 1.1 millions non-English git repos demonstrate that this is a large scale reality. Here unicode is relevant (after all, there must be a reason for new languages such as Swift, C# and others to accept unicode identifiers). Ironically, due to the historical problems, many keep using transliteration (e.g. in German ae instead of ä or, more ambiguous: in French e instead of é).
So, still nowadays, a significant number of people have kept in mind that there was something, and just continue to promote ASCII, although Unicode identifiers are now largely supported.
Edit: All unicode chars are still not equal!
Accepting unicode characters in an identifier does not ensure that all unicode characters are equally treated in all languages that claim to support it.
Unicode characters are categorized into classes, for example spacing, punctuation, letters (aka writing alphabet) and others. Some quick experimentation on my mac:
- Swift, C++ and C# interpret spacing characters properly as token separator, whereas python considers it as an error.
- Emojis are not letters. C++ and Swift accept them in an identifier, but python and C# don't (Example:
t🙂w) - Characters of class "letter" are generally accepted in an identifier in all my tests in the four languages. But some Egyptian hieroglyphs are not recognized as letter by C# (Examples:
t𓀉w,t丳w,téw, the first being the problematic one) - And good news for the mathematicians among us,
πis a valid identifier in all these languages ;-)
The big problem is seeing two identifiers, and determining just by looking whether they are the same. Capital Latin, Cyrillic and Greek characters for example often look the same. Chinese characters I just couldn’t recognise. And I think Swift allows certain white space characters in identifiers, other languages might as well. Imagine having an identifier that is literally invisible.
Having to use a different keyboard would be insane. On Mac keyboards, there are a few dozen non-ASCII characters easily available (I can type them without thinking), so anything in this set I would be fine, anything outside I wouldn’t. In one case I defined a ≈≈ operator (== didn’t work for some reasons). Other OSs have different easily accessible characters.
So I’m not totally against it, but be reasonable.
@Greg, the compilers have absolutely no problem with unicode characters. It's the programmer. The compiler can easily distinguish between three different white space characters, or defining an m-dash or n-dash as an operator instead of the hyphen. It's me who has the problem.
PS. Unfortunately Swift doesn't support an operator named ² or ³, so you can't write y = x³ or y = (a+b)². Well, you can write the first one because x³ is a valid identifier, so you can write
let x = 5
let x² = x*x
let x³ = x²*x
There are even Chinese people in the US. They name their children 李某. Would that be a problem? No. Some systems support these names, some use versions converted to ASCII through romanization (李某 → pinyin Lǐ Mǒu → Li Mou). The only non-ASCII character in X Æ A-12, Æ, is used e.g. in Danish names like Ægidius, converted to ASCII Aegidius.
A good example of such conversion is machine-readable passports: the last two lines on every passport contains only characters A–Z, 0–9 and the filler character <. For passports, every name in the world gets converted to ASCII.
Sometimes using mere ASCII may be more problematic than non-ASCII characters that are easily encoded in UTF-8:
Christopher Null is a real person who has reported some problems with his ASCII name.
My name is Logger, Startkeylogger.
Little Bobby Tables has a name as unique as the little baby Musk. This Robert doesn't have a social security number as he's a fictional character, but for real people the name isn't an obstacle.
I don't live in the US but I live in a street that has (if written correctly) 2 German Umlauts in their name.
I never had a problem with my passport or with US authorities, visa waiver programs or anything that concerns the US government. They will accept it either the way it is or a "translation" into non-Umlauts (which even exists in Germany as a legal way to write the same word without Umlauts).
I have no experience with SSN or unemployment systems in the US, but I have no reason to believe they will behave differently from the civil administration I did come in contact with that handled it without blinking.
Now private businesses are a whole other can of worms. It has become a lot better in the last 3-5 years, but I had to fit a lot of square pegs through round holes to use my address in the US. For example, I own a credit card from a major US brand. And one would assume you can use that to pay online, right? My credit card is no different than your credit card. Well, yes and no. One will need to give the address to the merchant that in turn will send it with the card data to the card processor to reach a higher confidence that it was actually me doing the purchase, not someone who scammed just my card number. So the crappy merchant website would not accept umlauts. No problem, use the other accepted spelling. Then the merchant would accept it, but the card processor would barf and say "no sir, that's not their correct address, as written down here.". I think I dropped 25% of my purchases online because the merchants were literally to stupid to draw my money from a major US brand credit card.
Another example: bring an App into the Apple store. That was only 3 years ago. Apple is a major international company, surely they would not screw it up, right? Well, they did not. Directly. I could create an account. But they have a partner where you need to be registered to be accepted as a company. Guess who had never heard of Umlauts? I needed to talk to third level support (actually a developer that had database access) to get that done because even their own internal support interface would not let them.
So... I think you will be fine with all official, administrative tasks, but don't expect an easy life. Life is not just death and taxes, if you want to do something in between, maybe something that's fun or makes money, it will be a lot easier if your name is "Jake Brown" than "Jörg Oßten".