InDesign uses Perl-compatible regular expressions (pcre). Getting a Unicode character into the replacement string is done by \x{XXXX} where XXXX is the hexadecimal character code:
2\x{2009}$5
But in general you can replace by any character you can type. Just put actual thin spaces into your search-and-replace dialog:
3 $5
You can use your OS's utilities to grab the thin space from the list of available characters, for Windows it's the "Character Map" tool, where the thin space can be found in the "General Punctuation" Unicode sub-range. Searching for "thin space" works as well. MacOS has the "Character Viewer", which can do the same thing.
If you're not worried about performance, you can replace all the manual work by using the <regex> facilities, which results in a considerable reduction of code to test and maintain.
#include <regex>
source = std::regex_replace(source, std::regex("€"), "\x80");
I would still keep it in a separate function to make it easy to change the implementation afterwards.
You don't need the std::vector<std::uint8_t> objects at all. You can use the input std::string objects directly.
Also, the code in the while loop needs to be updated for the following issues:
Make sure to capture the return value opf
source.erase. If you don't the iterator is invalid.To avoid infinite loop, use
itras the first argument tostd::search.Update
itrinside the loop appropriately to avoid an infinite loop.
void replaceAllOccurences(std::string& source,
const std::string& replaceFrom,
const std::string& replaceTo)
{
std::string::iterator itr = source.begin();
while((itr = std::search(itr, source.end(), replaceFrom.begin(), replaceFrom.end())) != source.end())
{
itr = source.erase(itr, itr + replaceFrom.size());
// itr is going be invalid after insert. Keep track of its
// distance from begin() so we can update itr after insert.
auto dist = std::distance(source.begin(), itr);
source.insert(itr, replaceTo.begin(), replaceTo.end());
// Make itr point to the character 1 past what got replaced.
// This will avoid infinite loop incase the first character of
// replaceTo is the same as the character being replaced.
itr = std::next(source.begin(), dist+1);
}
}
c# - Replace Unicode character "�" with a space - Stack Overflow
Regex unicode chars in C++ - Stack Overflow
find and replace unicode by regex
c# - replace unicode character - Stack Overflow
Using String.Replace:
Use a simple String.Replace().
I've assumed that the only characters you want to remove are the ones you've mentioned in the question: � and you want to replace them by a normal space.
string text = "imp�ortant";
string cleaned = text.Replace('\u00ef', ' ')
.Replace('\u00bf', ' ')
.Replace('\u00bd', ' ');
// Returns 'imp ortant'
Or using Regex.Replace:
string cleaned = Regex.Replace(text, "[\u00ef\u00bf\u00bd]", " ");
// Returns 'imp ortant'
Try it out: Dotnet Fiddle
Define a range of ASCII characters, and replace anything that is not within that range.
We want to find only Unicode characters, so we will match on a Unicode character and replace.
Regex.Replace("This is my te\uFFFDxt from csv file", @"[^\u0000-\u007F]+", " ")
The above pattern will match anything that is not ^ in the set [ ] of this range \u0000-\u007F (ASCII characters (everything past \u007F is Unicode)) and replace it with a space.
Result
This is my te xt from csv file
You can adjust the range provided \u0000-\u007F as needed to expand the range of allowed characters to suit your needs.
i have several strings of characters that have unicode at the end of them. I would like to do a find with regex and replace the unicode with the specified characters.
30May19Bel© 13Jun18Bel¬ 24Aug17Sar«
I would like to do a find and replace with regex
for example [0-9]{2}[a-zA-Z]{3}[0-9]{2}[a-zA-Z]{3}©
$path = 'C:temp\uni.txt'
$original_file = C:temp\uni.txt"
$destination_file = "C:temp\update.txt"
(Get-Content $original_file) | Foreach-Object {
$_ -replace '¨', '1'`
-replace '©', '2'`
-replace 'ª', '3'`
-replace '«', '4'`
-replace '¬', '5'`
-replace '°', '6'`
} | Set-Content $destination_fileand replace it with
30May19Bel2 13Jun18Bel5 24Aug17Sar4
This may be a caveman approach, and I know nothing about regex. If there is a better way, i am unaware of it, but totally open to someone dropping the egg of knowledge on me. It just seemed like the easiest way was to create a Rosetta Stone of sorts. Thanks, Rogue
So you've got some malformed Unicode escapes in the string and you want to fix them by removing any whitespace after the 0. That's simple enough:
jData = Regex.Replace(jData, @"(\\u0)\s+(\w+)", "$1$2");
The hardest part of all this is figuring out what all the backslashes are supposed to mean. C# can helps you with that supports an alternative string literal syntax for verbatim string, the only character that you have to escape with a backslash is the backslash itself. (You have to escape quotation marks too, but you do that with another quote, i.e. "").
With that out of the way, the real reason I answered this question was to advise you not to use RegexOptions.Compiled. I'm sure you've heard many people say it makes the regex work faster. That's true, but it's an oversimplification. Read this article for a good discussion of this issue. Do yourself a favor and forget RegexOptions.Compiled even exists until you run into a problem you can't solve without it.
find: @"\\u0 "
replace: @"\\u0"
they are the same. Try it with an capital O or normal o
It is possible to use hex values in "sed".
echo "Ã" | hexdump -C
00000000 c3 83 0a |...|
00000003
Ok, that character is two byte combination "c3 83". Let's replace it with single byte "A":
echo "Ã" |sed 's/\xc3\x83/A/g'
A
Explanation: \x indicates for "sed" that a hex code follows.
Try setting LANG=C and then run it over the Unicode range:
echo "hi ☠ there ☠" | LANG=C sed "s/[\x80-\xFF]//g"