php - How do I remove Unicode Characters from String? - Stack Overflow
regex - python regular expression to remove u'some text' from the unicode output - Stack Overflow
To remove Unicode character from String in Java using REGEX - Stack Overflow
python - Regular Expression To Remove Unicode Characters From Json File Not Working - Stack Overflow
Note that you may have some strings that are delimited by u" " instead, if the string contains single quotes. Also, there may be escape sequences. ast.literal_eval can handle all that for you:
from ast import literal_eval
def convert(original):
try:
result = literal_eval(original)
if isinstance(result, unicode):
return result
except ValueError:
pass
return original
This should do it:
import re
re.sub("^u'(.*)'$",r'\1',"u'text'")
-> text
This will work on a single value string without leading/trailing characters (e.g. "u'text'").
If you want replace all occurences of u'text' within a string, you can do this instead:
re.sub("u'([^']*)'",r'\1',STRING)
For instance:
re.sub("u'([^']*)'",r'\1',"u'value1',u'value2',u'value3'")
-> value1,value2,value3
You can do this sequentially like below:
public static void main(final String args[]) {
String comment = "Good morning! \u2028\u2028I am looking to purchase a new Honda car as I\u2019m outgrowing my current car. I currently drive a Hyundai Accent and I was looking for something a little bit larger and more comfortable like the Honda Civic. May I know if you have any of the models currently in stock? Thank you! Warm regards Sandra";
// remove all non-ASCII characters
comment = comment.replaceAll("[^\\x00-\\x7F]", "");
// remove all the ASCII control characters
comment = comment.replaceAll("[\\p{Cntrl}&&[^\r\n\t]]", "");
// removes non-printable characters from Unicode
comment = comment.replaceAll("\\p{C}", "");
System.out.println(comment);
}
If you use replace, you will lost some characters, For example I'm will become Im. So the best thing is convert.
You can Convert Unicode to UTF-8.
byte[] byteComment = comment.getBytes("UTF-8");
String formattedComment = new String(byteComment, "UTF-8");
Unicode is defined as any code point greater than or equal to x100.
The Python regex for UTF- 8 / 32 Unicode Ranges using \uXXXX \UXXXXXXXX syntax is :
[\u0100-\uFFFF\U00010000-\U0010FFFF]+
https://regex101.com/r/1YUSng/1
________________________
If for Python, should ever need the UTF-16 one for surrogates, it is :
(?:[\u0100-\uD7FF\uE000-\uFFFF]|[\uD800-\uDBFF][\uDC00-\uDFFF]|\uD800-\uDBFF|\uDC00-\uDFFF)+
I am not sure about how to do this in Python, but here is how I did it in JavaScript. May be the same replace function works in Python too
'use strict'
const str1 = 'test\uD83E\uDD2Astring';
let result = str1.replace(/([^ -~]+)/g, '');
console.log(`the result1 is ${result}`);
and the result is
the result1 is teststring
I used this link. Please go over it and check if you need some more clarifications.
If this does not help you solve your problem, I am happy to delete this answer.
This expression will search for non-ASCII values:
[^\x00-\x7F]+
Tick off 'Search Mode = Regular expression', and click Find Next.
Source: Regex any ASCII character
Try "Find characters in range..."
In Notepad++, if you go to menu Search → Find characters in range → Non-ASCII Characters (128-255)...
...you can then step through the document to each non-ASCII character.
Be sure to tick off Wrap around if you want to loop in the document for all non-ASCII characters.
When you press Find it selects the character.
Then go to the Edit menu and pick Replace, and the "find" box will be filled with the current selection, which will be the character you found.
Then you can do the rest of the find/replace in the normal dialog.
You are using a gsub function with a regex pattern as the first argument. <U+FFFD> pattern matches <, 1 or more U symbols, and then a FFFD> sequence of chars.
It would work like this:
> str2 <- "торгово <UUUFFFD> производственн��я компания"
> gsub("<U+FFFD>", "", str2)
[1] "торгово производственн��я компания"
Use a mere literal string replacement:
> str <- "торгово производственн��я компания"
> gsub("\uFFFD", "", str, fixed=TRUE)
[1] "торгово производствення компания"
This worked best for me when applying this same concept to an entire data frame.
# Remove embedded unicode characters in the data frame
df <- df %>%
mutate(across(where(is.character),~ str_remove_all(.,"\\s*\u200b")))
[^\x00-\x7F] works fine, but, if you want to use a long character class like [^a-z0-9``~!@#$%^&*()-_=+[]{}\|;:'"<>,./?] you have to escape characters that have a special meaning (ie. -[]\ and add linebreak \r,\n.
Your regex becomes:
[^a-z0-9``~!@#$%^&*()\-_=+\[\]{}\\|;:'"<>,./?\r\n]
# ^ ^ ^ ^ ^^^^
- Ctrl+H
- Find what:
[^a-z0-9``~!@#$%^&*()\-_=+\[\]{}\\|;:'"<>,./?\r\n]+$But, again,[^\x00-\x7F]works fine and is more readable - Replace with:
LEAVE EMPTY - check Wrap around
- check Regular expression
- Replace all
Result for given example:
0123456789`~!@#$%^&*()-_=+[]{}\|;:'"<>,./?
abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
If you are agnostic to the solution and not fixed to Notepad++ you could install bash for Win 10, as I showed here https://superuser.com/a/1252271/715210 (sorry I always come back to your questions with Linux workarounds ;) )
I would have a solution, where you unfortunately also will loose the apostrophe '
- open bash for Windows over start menu
- Go to the folder, where your file is located with
cd /mnt/c/path/folder(the drive C: is on /mnt/c) If your file is named foo.txt you could generate a file bar.txt with this command:
cat foo.txt | tr -cd '[:alnum:]\n\r~!@#$%^&*()-_=+{}\|;:<>,./?"`' | sed '/^$/d' > bar.txt
Explanation of the parts:
cat foo.txt outputs the text file and with the pipe | the output is redirected to the commande tr -cd which removes every char, which is not in the list after betwenn '...'. Followed by a pipe tosedto remove the empty lines. Last but not least with> bar.txt` we redirect the output to the file bar.txt
Thanks to:
- https://stackoverflow.com/a/20007549/7311363 for the :alnum: tip
- https://unix.stackexchange.com/a/48568/223965 for the apostrophe explanation
I've been trying to understand regex for discord. I've tried this so far. I'm trying to filter out everything but normal characters and the few special characters on keyboards if anyone has a dummies for regex guide please send it my way.
^[a-zA-Z0-9:.,?!@]