I wanted to url encode a python string and got exceptions with hebrew strings. I couldn't fix it and started doing some guess oriented programming. Finally, doing mystr = mystr.encode("utf8") before More on stackoverflow.com

Communities for your favorite technologies. Explore all Collectives · Stack Overflow for Teams is now called Stack Internal. Bring the best of human thought and AI automation together at your work More on stackoverflow.com

Whether you need to "support" utf-8/unicode depends very much on the application, all the way down to the micro level. For just copying a string/moving a buffer, you're probably never going to write utf-8 aware functions. But when comparing strings, you probably should. And when displaying strings, you may (depending on how much you're relying on UI libraries) have to deal with grapheme clusters, right-to-left rendering, etc. What has been most helpful to me in understanding which kind of processing is appropriate for what application, is to have a firm grasp on the fundamental concepts. The first step of this, is to take the word 'character', and fully eradicate it from your programming vocabulary. It is simply too overloaded to be useful. Depending on context/speaker it could mean: Byte (a unit of memory, the 'char' type in POSIX C) Code unit (a property of the encoding) Code point (a Unicode concept independent of encoding) Grapheme cluster (a more visual concept) And I've probably forgotten a few. Grasping the differences between the concepts above will go a long way, especially if you adopt them, with precision, in your thinking and implementation. More on reddit.com

Everything in a computer is a number. If we want to have letters in computers, we need to all agree on what number corresponds to what letter, this is called "character encoding". The simplest encoding is called ASCII. In ASCII, A is 65, B is 66, & is 38. ASCII characters are stored using only 7 bits, which means there's only 27=128 characters possible. ASCII works fine for encoding basic English/Latin characters, but there's way more than 128 characters in the world! Unicode is the name of a system that supports up to 1,112,064 characters, which includes letters from every major alphabet, and lots of other characters like math symbols, emojis and more. But now there's the question of how do we store these characters, how big does the "number have to be". UTF-32 is a character encoding that uses one 32 bit number for each character. This makes a lot of sense, but it wastes a lot of space. Why do we need a number big enough to hold 1,112,064 values, when most of the time we're only going to use the first 128 values. UTF-8 is a "variable length encoding". That means a lot of the time, each character only takes up 8 bits, but it can expand to up to 32 bits if necessary. This system can support any Unicode character without wasting space, which has made it the most popular character encoding. Now as for how this relates to webpages, when your browser gets a webpage from a server, it needs to know which encoding to use. If you don't specify anything, most browsers will default to ASCII. This is usually fine, since all characters required for html like < can be written in ASCII, and you can add special Unicode characters to your HTML by writing things like ♠. But if you want to include Unicode characters right in your code, then the file will be saved as UTF-8, and you will need to include a header or meta tag to tell your browser the right way to read your file. Hope that was helpful! More on reddit.com