Regarding searching by UTF-16 code

To search by Unicode codepoints using UTF-16 you'd use \x{FEC1}, and it works whether the file is encoded with UTF-8 or UTF-16.

Bear in mind you wouldn't need to search by the UTF-8 code, because you can search by the UTF-16 code. But to address the part of your question that asks how do you search for that character by the UTF-8 code...

Regarding searching by UTF-8 code

You can't. Well, you sort of can, but it's a hideous hack and you really shouldn't.

The obvious thing to try would be to search for \xef\xbb\x81 in your UTF-8 encoded document, but that doesn't work. (Note there's no {} here: Notepad++ expects either \xNN for 2 hex digits, or \x{NNNN} for 4 hex digits). That's because Notepad++ doesn't actually search for byte values, it searches for Unicode codepoints. So you can search for the codepoint U+FEC1, but not for the UTF-8 bytes 0xEF 0xBB 0x81, because Notepad++ "hides" the encoding details from you. (Because in nearly every scenario, someone editing a text file will care far more about finding the actual character than about finding the UTF-8 bytes.)

There's another trick you might try, which is to take that UTF-8 encoded file and choose the Encoding → Encode in ANSI menu option, at which point ﻁﻁﻉﻁﻉﻁﻉ appears to become ï»ï»ï»‰ï»ï»‰ï»ï»‰. (I say "appears to become" rather than "becomes" because... well, read on.) This is because it has taken the UTF-8 text of your file, and reinterpreted it as "ANSI" (which is a terrible encoding name because it's completely wrong, and should really be called "Windows-1252", but that's a different question). (By the way, the reason that ﻁﻁﻉﻁﻉﻁﻉ looks backwards in my text than the way it does in your screenshot: that's because Notepad++ doesn't care that Arabic is written right-to-left, so it shows the characters left-to-right in the order they were pasted into the file. But your browser does care about presenting Arabic in proper right-to-left order, the first two letters of that string (ﻁﻁ) appear on the right-hand side of the string, not on the left-hand side as they seem to in Notepad++). Digressions aside, here's why this will be helpful. In the "ANSI" (really Windows-1252) encoding, each byte is a single character, and so now you're going to be able to search by individual bytes. Now, if you search for \xef\xbb\x81 (which doesn't need to be a regular expression, just an "Extended" search), it will find the characters. Sort of. It will look like it's highlighting the two characters ï», but it's really highlighting three characters: ï, », and an "invisible" 0x81 character that doesn't really exist. (Because there is no character at the 0x81 point in Windows-1252 encoding: see for yourself.) And now you see why I said "appears to become" -- because your UTF-8 encoded text has really become ï»_ï»_ﻉï»_ﻉï»_ﻉ, where _ represents an "invisible" character that doesn't officially exist in the Windows-1252 codepage. Anyway, now that you've found the sequence of three characters with the byte values 0xEF, 0xBB, and 0x81 in Windows-1252, and Notepad++ has highlighted them, you can choose the Encoding → Encode in UTF-8 menu option, and your text will convert itself back to UTF-8, while Notepad++ will keep the highlight in the same place -- and thus, you'll find that one ﻁ character has been highlighted.

So why do I say that you really shouldn't do this? Because the only reason that it works is that Notepad++ didn't do the right thing when you switched codepages. The right thing to do when you find a missing character is to complain, or insert a character like the Unicode replacement character � (or a simple ? if you're in a legacy codepage that doesn't have � in it), or do something so that the user will know they had an invalid character in their text. Errors should never be silently ignored, and having a 0x81 value in Windows-1252 text is an error. The only reason this trick works is because Notepad++ does the wrong thing with invalid characters (that is, it ignores them). So you really shouldn't rely on this trick: with any update to Notepad++, it could change its undocumented (and wrong) behavior, and start putting proper replacement characters in wrongly-encoded text, at which point this trick would fail. Stick to searching for real Unicode codepoints, and you'll be much better off.

By the way, the reason why your original attempt ([\uFEC1]) failed is because, according to Notepad++'s regular expression syntax, \u means "an uppercase letter". (Remember that in regular expressions, brackets represent "any of these characters"). The docs further say, "See note about lower case [sic] letters," and the note about lowercase letters says "this will fall back on "a word character" if the "Match case" search option is off." As it is in your screenshot. Therefore, the regex [\uFEC1] is searching for "any word character, or F, or E, or C, or 1" -- which matches every single character in your sample text.

Phew, that turned out to be a very long answer for what I said would be "very simple". I hope this helps you understand Unicode a bit better; if so, the hour I spent typing this up will have been worth it.

Assuming your script does know the correct encoding of your text snippet than that should be the regular expression to find all Non-ASCII charactres:

[^\x00-\x7F]+

see here: https://stackoverflow.com/a/20890052/1144966 and https://stackoverflow.com/a/8845398/1144966

Also, the base-R tools package provides two functions to detect non-ASCII characters:

tools::showNonASCII()
tools::showNonASCIIfile()

Emacs

C-x = (M-x what-cursor-position) shows code point information about the current character at the bottom of the screen, for example

Char: И (1048, #o2030, #x418, file ...) point=7 of 8 (75%) column=0

C-u C-x = opens a window with more information, including the code point, the byte representation, metadata about the Unicode character, what font is used to display the character, etc.

             position: 7 of 8 (75%), column: 0
            character: И (displayed as И) (codepoint 1048, #o2030, #x418)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x0418
               script: cyrillic
               syntax: w    which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #xD0 #x98
            file code: #xD0 #x98 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-DAMA-Ubuntu Mono-normal-normal-normal-*-17-*-*-*-m-0-iso10646-1 (#x2CB)
         Unicode data:
                 Name: CYRILLIC CAPITAL LETTER I
             Category: Letter, Uppercase
      Combining class: Lu
        Bidi category: Lu
             Old name: CYRILLIC CAPITAL LETTER II
            Lowercase: и

Character code properties: customize what to show
  name: CYRILLIC CAPITAL LETTER I
  old-name: CYRILLIC CAPITAL LETTER II
  general-category: Lu (Letter, Uppercase)
  decomposition: (1048) ('И')

[back]

Command line

The unicode utility (available in some distributions such as Debian/Ubuntu/…, or with pip3 install unicode) shows information about one or more Unicode characters. Note that if you copy-paste from an editor, that editor may encode the clipboard differently from the file.

$ unicode И
U+0418 CYRILLIC CAPITAL LETTER I
UTF-8: d0 98 UTF-16BE: 0418 Decimal: И Octal: \02030
И (и)
Lowercase: 0438
Category: Lu (Letter, Uppercase)
Bidi: L (Left-to-Right)

grep is the wrong tool for the job.

You see the � U+FFFD REPLACEMENT CHARACTER not because it’s literally in the file content, but because you looked at a binary file with a tool that is supposed to handle only text-based input. The standard way to handle invalid input (i.e., random binary data) is to replace everything that is not valid in the current locale (most probably UTF-8) with U+FFFD before it hits the screen.

That means it is very likely that a literal \xEF\xBF\xBD (the UTF-8 byte sequence for the U+FFFD character) never occurs in the file. grep is completely right in telling you, there is none.

One way to detect whether a file contains some unknown binary is with the file(1) command:

$ head -c 100 /dev/urandom > rubbish.bin
$ file rubbish.bin
rubbish.bin: data

For any unknown file type it will simply say data. Try

$ file out.txt | grep '^out.txt: data$'

to check whether the file really contains any arbitrary binary and thus most likely rubbish.

If you want to make sure that out.txt is a UTF-8 encoded text file only, you can alternatively use iconv:

$ iconv -f utf-8 -t utf-16 out.txt >/dev/null

You can use ANSI-C quoting provided by your shell, to replace backslash-escaped characters as specified by the ANSI C standard. This should work for any command, not just grep, in shells like Bash and Zsh:

grep $'\u16A0'

For some more complex examples, you might refer to this related question and its answers.

See “How to detect the character encoding of a text-file?” or “How to reliably guess the encoding [...]?”

• UTF-8 can be detected with validation. You can also look for the BOM EF BB BF, but don't rely on it.
• UTF-16 can be detected by looking for the BOM.
• UTF-32 can be detected by validation, or by the BOM.
• Otherwise assume the ANSI code page.

Our codebase doesn't include any non-ASCII chars. I will try to grep for the BOM in files in our codebase. Thanks for the clarification.

Well that makes things a lot simpler. UTF-8 without non-ASCII chars is ASCII.

Assuming you have your locale set to UTF-8 (see locale output), this works well to recognize invalid UTF-8 sequences:

grep -axv '.*' file.txt

Explanation (from grep man page):

• -a, --text: treats file as text, essential prevents grep to abort once finding an invalid byte sequence (not being utf8)
• -v, --invert-match: inverts the output showing lines not matched
• -x '.*' (--line-regexp): means to match a complete line consisting of any utf8 character.

Hence, there will be output, which is the lines containing the invalid not utf8 byte sequence containing lines (since inverted -v)