This is why implicit conversion of byte strings to Unicode strings was removed in Python 3.
You're almost there, with the #coding line at the start of your file. Just one tiny change to turn your test character into a Unicode string:
if u'⸣' not in myvariable:
newvariable = 100.0
You might have trouble with that particular character as I did on my system, so you can use the equivalent escape sequence instead:
if u'\u2e23' not in myvariable:
newvariable = 100.0
Answer from Mark Ransom on Stack OverflowThis is why implicit conversion of byte strings to Unicode strings was removed in Python 3.
You're almost there, with the #coding line at the start of your file. Just one tiny change to turn your test character into a Unicode string:
if u'⸣' not in myvariable:
newvariable = 100.0
You might have trouble with that particular character as I did on my system, so you can use the equivalent escape sequence instead:
if u'\u2e23' not in myvariable:
newvariable = 100.0
You can declare the unicode as eg: var = u'e' and do the following operation var.find('a') to find the character in the unicode variable.
Hope this works !!
Python - How can I do a string find on a Unicode character that is a variable? - Stack Overflow
How to check Unicode char in python - Stack Overflow
python - Searching String for a Unicode Character - Stack Overflow
python - Find out the unicode script of a character - Stack Overflow
Since I'm using a variable now, how do I indicate the string represented by the variable is Unicode?
By defining it as a Unicode string in the first place.
zzz = u"foo"
Or, if you already have a string in some other encoding, by converting it to Unicode (the original encoding must be specified if the string is non-ASCII).
zzz = unicode(zzz, encoding="latin1")
Or by using Python 3 where all strings are Unicode.
zzz as defined in your post is a plain str object, not a unicode object, so there is no way to indicate that it is something it actually isn't. You can convert the str object to a unicode object, though, by specifying an encoding:
s.find(zzz.decode("utf-8"))
Substitue utf-8 by whatever encoding the string is encoded in.
Note that in your example
zzz = '\u0101'
zzz is a plain string of length 6. There is no easy way to fix this wrong string literal afterwards, except for hacks along the lines of
ast.literal_eval("u'" + zzz + "'")
for x in body:
if ord(x) > 127:
# character is *not* ASCII
This works if you have a Unicode string. If you just want to detect if the string contains a non-ASCII character it also works on a UTF-8 encoded byte string.
Update for Python 3: the above still works on Unicode strings, but ord no longer works for byte strings. But that's OK, because indexing into a byte string already returns an integer - no conversion necessary! The code becomes even simpler, especially if you combine it with the any function:
if any(x > 127 for x in body):
# string is *not* ASCII
Not Sure, whats your exact requirement is
>>> u'aあä'.encode('ascii', 'ignore')
'a'
I Found this at Python: Convert Unicode to ASCII without errors
I was hoping someone's done it before, but apparently not, so here's what I've ended up with. The module below (I call it unicodedata2) extends unicodedata and provides script_cat(chr) which returns a tuple (Script name, Category) for a unicode char. Example:
# coding=utf8
import unicodedata2
print unicodedata2.script_cat(u'Ф') #('Cyrillic', 'L')
print unicodedata2.script_cat(u'の') #('Hiragana', 'Lo')
print unicodedata2.script_cat(u'★') #('Common', 'So')
The module: https://gist.github.com/2204527
It seems to me that the Python unicodedata module contains tools for accessing the main file in the Unicode database but nothing for the other files: “The data in this database is based on the UnicodeData.txt file”
The script information is in the Scripts.txt file. It is of relatively simple format (described in UAX #44) and not horribly large (131 kilobytes), so you might consider parsing it in your program. Note that in the Unicode classification, there’s the “Common” script that contains characters used in different scripts, like punctuation marks.
In Python 3, all strings are sequences of Unicode characters. There is a bytes type that holds raw bytes.
In Python 2, a string may be of type str or of type unicode. You can tell which using code something like this:
def whatisthis(s):
if isinstance(s, str):
print "ordinary string"
elif isinstance(s, unicode):
print "unicode string"
else:
print "not a string"
This does not distinguish "Unicode or ASCII"; it only distinguishes Python types. A Unicode string may consist of purely characters in the ASCII range, and a bytestring may contain ASCII, encoded Unicode, or even non-textual data.
How to tell if an object is a unicode string or a byte string
You can use type or isinstance.
In Python 2:
>>> type(u'abc') # Python 2 unicode string literal
<type 'unicode'>
>>> type('abc') # Python 2 byte string literal
<type 'str'>
In Python 2, str is just a sequence of bytes. Python doesn't know what
its encoding is. The unicode type is the safer way to store text.
If you want to understand this more, I recommend http://farmdev.com/talks/unicode/.
In Python 3:
>>> type('abc') # Python 3 unicode string literal
<class 'str'>
>>> type(b'abc') # Python 3 byte string literal
<class 'bytes'>
In Python 3, str is like Python 2's unicode, and is used to
store text. What was called str in Python 2 is called bytes in Python 3.
How to tell if a byte string is valid utf-8 or ascii
You can call decode. If it raises a UnicodeDecodeError exception, it wasn't valid.
>>> u_umlaut = b'\xc3\x9c' # UTF-8 representation of the letter 'Ü'
>>> u_umlaut.decode('utf-8')
u'\xdc'
>>> u_umlaut.decode('ascii')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
To include Unicode characters in your Python source code, you can use Unicode escape characters in the form \u0123 in your string. In Python 2.x, you also need to prefix the string literal with 'u'.
Here's an example running in the Python 2.x interactive console:
>>> print u'\u0420\u043e\u0441\u0441\u0438\u044f'
Россия
In Python 2, prefixing a string with 'u' declares them as Unicode-type variables, as described in the Python Unicode documentation.
In Python 3, the 'u' prefix is now optional:
>>> print('\u0420\u043e\u0441\u0441\u0438\u044f')
Россия
If running the above commands doesn't display the text correctly for you, perhaps your terminal isn't capable of displaying Unicode characters.
These examples use Unicode escapes (\u...), which allows you to print Unicode characters while keeping your source code as plain ASCII. This can help when working with the same source code on different systems. You can also use Unicode characters directly in your Python source code (e.g. print u'Россия' in Python 2), if you are confident all your systems handle Unicode files properly.
For information about reading Unicode data from a file, see this answer:
Character reading from file in Python
Print a unicode character in Python:
Print a unicode character directly from python interpreter:
el@apollo:~$ python
Python 2.7.3
>>> print u'\u2713'
✓
Unicode character u'\u2713' is a checkmark. The interpreter prints the checkmark on the screen.
Print a unicode character from a python script:
Put this in test.py:
#!/usr/bin/python
print("here is your checkmark: " + u'\u2713');
Run it like this:
el@apollo:~$ python test.py
here is your checkmark: ✓
If it doesn't show a checkmark for you, then the problem could be elsewhere, like the terminal settings or something you are doing with stream redirection.
Store unicode characters in a file:
Save this to file: foo.py:
#!/usr/bin/python -tt
# -*- coding: utf-8 -*-
import codecs
import sys
UTF8Writer = codecs.getwriter('utf8')
sys.stdout = UTF8Writer(sys.stdout)
print(u'e with obfuscation: é')
Run it and pipe output to file:
python foo.py > tmp.txt
Open tmp.txt and look inside, you see this:
el@apollo:~$ cat tmp.txt
e with obfuscation: é
Thus you have saved unicode e with a obfuscation mark on it to a file.
Instead of scraping entirely via xpath like I have in the past, I decided to pull the data out of a clearly visible var in a <script> element and write a function to parse it as a dictionary recursively*.
My problem is that many characters are escaped, so instead of https:// I have https:\u002F\u002F, literally -- 12 characters instead of those 2 forward slashes, the \u**** sequence doesn't stand for the frontslashes. I have this problem in body text too, and because there are lots of diacritical marks I can't get away with replacing them bluntly. I need to replace these substrings with the corresponding characters, so \u002F -> /, \u00C9 -> é, etc.
It seems like there should be an obvious and not-ridiculous solution for this, but I don't know what it is and googling turns up different problems. Is this something I need to anticipate at the scraping stage? I'm using the requests and html modules if it matters.
*Which is dumb because it was already formatted just like a dict, so I could probably just have saved the raw text in a .py and imported it as a module
You'll want to use the unichr() builtin function:
for i in range(1000,1100):
print i, unichr(i)
Note that in Python 3, just chr() will suffice.
Use unichr:
s = unichr(i)
From the documentation:
unichr(i)Return the Unicode string of one character whose Unicode code is the integer i. For example, unichr(97) returns the string u'a'.