Encode each value in the list to a string:
[x.encode('UTF8') for x in EmployeeList]
You need to pick a valid encoding; don't use str() as that'll use the system default (for Python 2 that's ASCII) which will not encode all possible codepoints in a Unicode value.
UTF-8 is capable of encoding all of the Unicode standard, but any codepoint outside the ASCII range will lead to multiple bytes per character.
However, if all you want to do is test for a specific string, test for a unicode string and Python won't have to auto-encode all values when testing for that:
u'1001' in EmployeeList.values()
Answer from Martijn Pieters on Stack OverflowEncode each value in the list to a string:
[x.encode('UTF8') for x in EmployeeList]
You need to pick a valid encoding; don't use str() as that'll use the system default (for Python 2 that's ASCII) which will not encode all possible codepoints in a Unicode value.
UTF-8 is capable of encoding all of the Unicode standard, but any codepoint outside the ASCII range will lead to multiple bytes per character.
However, if all you want to do is test for a specific string, test for a unicode string and Python won't have to auto-encode all values when testing for that:
u'1001' in EmployeeList.values()
[str(x) for x in EmployeeList] would do a conversion, but it would fail if the unicode string characters do not lie in the ascii range.
>>> EmployeeList = [u'1001', u'Karick', u'14-12-2020', u'1$']
>>> [str(x) for x in EmployeeList]
['1001', 'Karick', '14-12-2020', '1$']
>>> EmployeeList = [u'1001', u'करिक', u'14-12-2020', u'1$']
>>> [str(x) for x in EmployeeList]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
Every codepoint has a name, so you are effectively asking for the Unicode standard list of codepoint names (as well as the *list of name aliases, supported by Python 3.3 and up).
Each Python version supports a specific version of the Unicode standard; the unicodedata.unidata_version attribute tells you which one for a given Python runtime. The above links lead to the latest published Unicode version, replace UCD/latest in the URLs with the value of unicodedata.unidata_version for your Python version.
Per codepoint, the unicodedata.name() function can tell you the official name, and unicodedata.lookup() gives you the inverse (name to codepoint).
If you want a list of all unicode character names, consider downloading the Unicode Character Database.
It is included in the base repositories of many linux distributions (ex. "unicode-ucd" on RHEL).
The package includes NamesList.txt, which contains the exhaustive list of unicode character names.
Caution: NamesList.txt need some times to be downloaded (size > 1.5 MB).
Example:
21FE RIGHTWARDS OPEN-HEADED ARROW
21FF LEFT RIGHT OPEN-HEADED ARROW
@@ 2200 Mathematical Operators 22FF
@@+
@ Miscellaneous mathematical symbols
2200 FOR ALL
= universal quantifier
2201 COMPLEMENT
x (latin letter stretched c - 0297)
2202 PARTIAL DIFFERENTIAL
2203 THERE EXISTS
= existential quantifier
2204 THERE DOES NOT EXIST
: 2203 0338
2205 EMPTY SET
= null set
* used in linguistics to indicate a null morpheme or phonological "zero"
x (latin capital letter o with stroke - 00D8)
x (diameter sign - 2300)
~ 2205 FE00 zero with long diagonal stroke overlay form
try applying unicode to its elements
>>> lst=['Justice league', 'Avenger']
>>> map(unicode,lst)
[u'Justice league', u'Avenger']
Obviously we are talking about Python 2 only, since literal strings are unicode by default in Python 3
Have a look at this Q&A on the same subject
Use the built in unicode function with a list comprehension:
x = ['Justice league', 'Avenger']
answer = [unicode(item) for item in x]
You can decode those Unicode escape sequences with .decode('unicode-escape'). However, .decode is a bytes method, so if those sequences are text rather than bytes you first need to encode them into bytes. Alternatively, you can (probably) open your CSV file in binary mode in order to read those sequences as bytes rather than as text strings.
Just for fun, I'll also use unicodedata to get the names of those emojis.
import unicodedata as ud
emojis = [
'\\U0001F600',
'\\U0001F601',
'\\U0001F602',
'\\U0001F923',
]
for u in emojis:
s = u.encode('ASCII').decode('unicode-escape')
print(u, ud.name(s), s)
output
\U0001F600 GRINNING FACE 😀
\U0001F601 GRINNING FACE WITH SMILING EYES 😁
\U0001F602 FACE WITH TEARS OF JOY 😂
\U0001F923 ROLLING ON THE FLOOR LAUGHING 🤣
This should be much faster than using ast.literal_eval. And if you read the data in binary mode it will be even faster since it avoids the initial decoding step while reading the file, as well as allowing you to eliminate the .encode('ASCII') call.
You can make the decoding a little more robust by using
u.encode('Latin1').decode('unicode-escape')
but that shouldn't be necessary for your emoji data. And as I said earlier, it would be even better if you open the file in binary mode to avoid the need to encode it.
1. keeping your csv as-is:
it's a bloated solution, but using ast.literal_eval works:
import ast
s = '\\U0001F600'
x = ast.literal_eval('"{}"'.format(s))
print(hex(ord(x)))
print(x)
I get 0x1f600 (which is correct char code) and some emoticon character (😀). (well I had to copy/paste a strange char from my console to this answer textfield but that's a console issue by my end, otherwise that works)
just surround with quotes to allow ast to take the input as string.
2. using character codes directly
maybe you'd be better off by storing the character codes themselves instead of the \U format:
print(chr(0x1F600))
does exactly the same (so ast is slightly overkill)
your csv could contain:
0x1F600
0x1F601
0x1F602
0x1F923
then chr(int(row[0],16)) would do the trick when reading it: example if one 1 row in CSV (or first row)
with open("codes.csv") as f:
cr = csv.reader(f)
codes = [int(row[0],16) for row in cr]