According to my interpretation of the implementation of unicode-escape and the unicode repr in the CPython 2.6.5 source, yes; the only difference between repr(unicode_string) and unicode_string.encode('unicode-escape') is the inclusion of wrapping quotes and escaping whichever quote was used.
They are both driven by the same function, unicodeescape_string. This function takes a parameter whose sole function is to toggle the addition of the wrapping quotes and escaping of that quote.
How to cast escaped unicode characters embeded into an string?
encoding - Python "string_escape" vs "unicode_escape" - Stack Overflow
How do convert unicode escape sequences to unicode characters in a python string - Stack Overflow
Escaping unicode strings in python - Stack Overflow
Sorry for the weird question.
I got a string like this one:
Some text\r\n word word word\u0022
I think those backslashed characters are escape sequences, and i need to convert/cast them to the character they represent.
For example, i think the \u0022 is the doble quotes ( " ), so i need to convert the string to this:
Some text\r\n word word word"
Is it possible to do this whithout having to replace every character manually (with replace() string method)?
I don't know if i could convert the breakline. In that particular case, there wouldn't be a problem if i just replace it with a space, but i need to cast every other character.
I hope you could understand what i mean. Thanks in advance and sorry for this weird and tricky question.
According to my interpretation of the implementation of unicode-escape and the unicode repr in the CPython 2.6.5 source, yes; the only difference between repr(unicode_string) and unicode_string.encode('unicode-escape') is the inclusion of wrapping quotes and escaping whichever quote was used.
They are both driven by the same function, unicodeescape_string. This function takes a parameter whose sole function is to toggle the addition of the wrapping quotes and escaping of that quote.
Within the range 0 β€ c < 128, yes the ' is the only difference for CPython 2.6.
>>> set(unichr(c).encode('unicode_escape') for c in range(128)) - set(chr(c).encode('string_escape') for c in range(128))
set(["'"])
Outside of this range the two types are not exchangeable.
>>> '\x80'.encode('string_escape')
'\\x80'
>>> '\x80'.encode('unicode_escape')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec canβt decode byte 0x80 in position 0: ordinal not in range(128)
>>> u'1'.encode('unicode_escape')
'1'
>>> u'1'.encode('string_escape')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: escape_encode() argument 1 must be str, not unicode
On Python 3.x, the string_escape encoding no longer exists, since str can only store Unicode.
Assuming Python sees the name as a normal string, you'll first have to decode it to unicode:
>>> name
'Christensen Sk\xf6ld'
>>> unicode(name, 'latin-1')
u'Christensen Sk\xf6ld'
Another way of achieving this:
>>> name.decode('latin-1')
u'Christensen Sk\xf6ld'
Note the "u" in front of the string, signalling it is uncode. If you print this, the accented letter is shown properly:
>>> print name.decode('latin-1')
Christensen SkΓΆld
BTW: when necessary, you can use de "encode" method to turn the unicode into e.g. a UTF-8 string:
>>> name.decode('latin-1').encode('utf-8')
'Christensen Sk\xc3\xb6ld'
I suspect that it's acutally working correctly. By default, Python displays strings in ASCII encoding, since not all terminals support unicode. If you actually print the string, though, it should work. See the following example:
>>> u'\xcfa'
u'\xcfa'
>>> print u'\xcfa'
Γa
The first one is a byte string:
>>> "\xF0\x9F\x8C\x80".decode('utf8')
u'\U0001f300'
The u"\ud83c\udf00" one is the UTF16 version (four digit unicode escape)
The u"\U0001F300" one is actual index of the codepoint.
But how do the numbers relate? This is the difficult question. It's defined by the encoding and there is no obvious relationship. To give you an idea, here is an example of "manually" encoding the codepoint at index 0x1F300 into UTF-8:
The cyclone character π has index 0x1f300 which falls into the range 0x00010000 - 0x001FFFFF. The template for this range is:
11110... 10...... 10...... 10......
Where you fill in the dots with the binary representation of the codepoint. I can't tell you why the template looks like that, it's just the utf-8 definition.
Here's the binary representation of our codepoint:
>>> u'π'
u'\U0001f300'
>>> unichr(0x1f300)
u'\U0001f300'
>>> bin(0x1f300)
'0b11111001100000000'
So if we take the string template and fill it up like this (with some leading zeros because there are more slots in the template than significant digits in our number) we get this:
11110... 10...... 10...... 10......
11110000 10011111 10001100 10000000
Now let's convert that back to hex
>>> 0b11110000100111111000110010000000
4036988032
>>> hex(4036988032)
'0xf09f8c80'
And there you have the UTF8 representation of the codepoint.
For UTF16 there is a similar magic recipe for your codepoint: 0x10000 is subtracted from the index, and then we pad with zeros to get a 20-bit binary representation. The first ten bits are added to 0xD800 to give the first 16-bit code unit. The last ten bits are added to 0xDC00 to give the second 16-bit code unit.
>>> bin(0x1f300 - 0x10000)[2:].rjust(20, '0')
'00001111001100000000'
>>> _[:10], _[10:]
('0000111100', '1100000000')
>>> hex(0b0000111100 + 0xd800)
'0xd83c'
>>> hex(0b1100000000 + 0xdc00)
'0xdf00'
And there's your UTF 16 version, i.e. the one with the lowercase \u escape.
As you can probably understand there may be no obvious numerical relationship between the hex digits in these representations, they are just different encodings of the same code point.
See Unicode Literals in Python Source Code
In Python source code, Unicode literals are written as strings prefixed with the βuβ or βUβ character:
u'abcdefghijk'. Specific code points can be written using the\uescape sequence, which is followed by four hex digits giving the code point. The\Uescape sequence is similar, but expects 8 hex digits, not 4.
In [1]: "\xF0\x9F\x8C\x80".decode('utf-8')
Out[1]: u'\U0001f300'
In [2]: u'\U0001F300'.encode('utf-8')
Out[2]: '\xf0\x9f\x8c\x80'
In [3]: u'\ud83c\udf00'.encode('utf-8')
Out[3]: '\xf0\x9f\x8c\x80'
\uhhhh --> Unicode character with 16-bit hex value
\Uhhhhhhhh --> Unicode character with 32-bit hex value
In Unicode escapes, the first form gives four hex digits to encode a 2-byte (16-bit) character code point, and the second gives eight hex digits for a 4-byte (32-bit) code point. Byte strings support only hex escapes for encoded text and other forms of byte-based data
Instead of scraping entirely via xpath like I have in the past, I decided to pull the data out of a clearly visible var in a <script> element and write a function to parse it as a dictionary recursively*.
My problem is that many characters are escaped, so instead of https:// I have https:\u002F\u002F, literally -- 12 characters instead of those 2 forward slashes, the \u**** sequence doesn't stand for the frontslashes. I have this problem in body text too, and because there are lots of diacritical marks I can't get away with replacing them bluntly. I need to replace these substrings with the corresponding characters, so \u002F -> /, \u00C9 -> Γ©, etc.
It seems like there should be an obvious and not-ridiculous solution for this, but I don't know what it is and googling turns up different problems. Is this something I need to anticipate at the scraping stage? I'm using the requests and html modules if it matters.
*Which is dumb because it was already formatted just like a dict, so I could probably just have saved the raw text in a .py and imported it as a module
The problem is with the string
"C:\Users\Eric\Desktop\beeline.txt"
Here, \U in "C:\Users... starts an eight-character Unicode escape, such as \U00014321. In your code, the escape is followed by the character 's', which is invalid.
You either need to duplicate all backslashes:
"C:\\Users\\Eric\\Desktop\\beeline.txt"
Or prefix the string with r (to produce a raw string):
r"C:\Users\Eric\Desktop\beeline.txt"
Typical error on Windows because the default user directory is C:\user\<your_user>, so when you want to pass this path as a string argument into a Python function, you get a Unicode error, just because the \u is a Unicode escape. If the next 8 characters after the \u are not numeric this produces an error.
To solve it, just double the backslashes: C:\\user\\<\your_user>...
This will ensure that Python treats the single backslashes as single backslashes.