Cleaner version:

>>> test_string = '1101100110000110110110011000001011011000101001111101100010101000'
>>> print ('%x' % int(test_string, 2)).decode('hex').decode('utf-8')
نقاب

Inverse (from @Robᵩ's comment):

>>> '{:b}'.format(int(u'نقاب'.encode('utf-8').encode('hex'), 16))
1: '1101100110000110110110011000001011011000101001111101100010101000'

"binary" isn't an encoding (character-set name). iconv needs an encoding name to do its job.

The file utility doesn't give useful information when it doesn't recognize the file format. It could be UTF-16 for example, without a byte-encoding-mark (BOM). notepad reads that. The same applies to UTF-8 (and head would display that since your terminal may be set to UTF-8 encoding, and it would not care about a BOM).

If the file is UTF-16, your terminal would display that using head because most of the characters would be ASCII (or even Latin-1), making the "other" byte of the UTF-16 characters a null.

In either case, the lack of BOM will (depending on the version of file) confuse it. But other programs may work, because these file formats can be used with Microsoft Windows as well as portable applications that may run on Windows.

To convert the file to UTF-8, you have to know which encoding it uses, and what the name for that encoding is with iconv. If it is already UTF-8, then whether you add a BOM (at the beginning) is optional. UTF-16 has two flavors, according to which byte is first. Or you could even have UTF-32. iconv -l lists these:

ISO-10646/UTF-8/
ISO-10646/UTF8/
UTF-7//
UTF-8//
UTF-16//
UTF-16BE//
UTF-16LE//
UTF-32//
UTF-32BE//
UTF-32LE//
UTF7//
UTF8//
UTF16//
UTF16BE//
UTF16LE//
UTF32//
UTF32BE//
UTF32LE//

"LE" and "BE" refer to little-end and big-end for the byte-order. Windows uses the "LE" flavors, and iconv likely assumes that for the flavors lacking "LE" or "BE".

You can see this using an octal (sic) dump:

$ od -bc big-end
0000000 000 124 000 150 000 165 000 040 000 101 000 165 000 147 000 040
         \0   T  \0   h  \0   u  \0      \0   A  \0   u  \0   g  \0    
0000020 000 061 000 070 000 040 000 060 000 065 000 072 000 060 000 061
         \0   1  \0   8  \0      \0   0  \0   5  \0   :  \0   0  \0   1
0000040 000 072 000 065 000 067 000 040 000 105 000 104 000 124 000 040
         \0   :  \0   5  \0   7  \0      \0   E  \0   D  \0   T  \0    
0000060 000 062 000 060 000 061 000 066 000 012
         \0   2  \0   0  \0   1  \0   6  \0  \n
0000072

$ od -bc little-end
0000000 124 000 150 000 165 000 040 000 101 000 165 000 147 000 040 000
          T  \0   h  \0   u  \0      \0   A  \0   u  \0   g  \0      \0
0000020 061 000 070 000 040 000 060 000 065 000 072 000 060 000 061 000
          1  \0   8  \0      \0   0  \0   5  \0   :  \0   0  \0   1  \0
0000040 072 000 065 000 067 000 040 000 105 000 104 000 124 000 040 000
          :  \0   5  \0   7  \0      \0   E  \0   D  \0   T  \0      \0
0000060 062 000 060 000 061 000 066 000 012 000
          2  \0   0  \0   1  \0   6  \0  \n  \0
0000072

Assuming UTF-16LE, you could convert using

iconv -f UTF-16LE// -t UTF-8// <input >output

You don't need a loop in your encode function

All the functions you're using are already vectorised so you can just as easily do:

encode2 <- function(message) {
    charToRaw(message) |>
        rawToBits() |>
        as.integer() |>
        rev()
}

x <- "some random string"
all(encode(x) == encode2(x))
# [1] TRUE

Build in handling of other locales

It's probably safer to use enc2utf8(), in case you're running in a different locale. As you explain in the comments you need to rev(), I would suggest something like:

encode3 <- function(message) {
    message |>
        enc2utf8() |>
        charToRaw() |>
        rawToBits() |>
        as.integer() |>
        rev()
}

message <- "café naïve — ∑ ∆ ∫ √ π λ 東京 😀 नमस्ते ✔ שלום ♥ ♦ ♣ ♠"
encoded <- encode3(message)
encoded
#  [1] 1 0 1 0 0 0 0 0 1 0 0 1 1 0 0 1 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 1 1 1 0 0 1 1 0 0 1 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 1 0 1 0 0 1 1 0 0 1 1 1 [83] 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 0 1 1 0 0 1 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 1 1 1 0 1 1 1 0 1 0 1 1 1 1 0 0 1 0 1 0 1 1 1 0 1 0 1 1 1 1 0 0 1 [165] 1 1 0 0 1 1 0 1 0 1 1 1 1 0 1 0 1 0 0 1 1 1 0 1 0 1 1 1 0 0 1 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 1 1 1 0 0 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 1 1 0 1 0 0 1 [247] 0 1 1 1 1 0 0 0 0 0 1 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 1 1 0 0 0 0 0 1 0 0 0 1 1 0 1 1 0 1 0 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 1 1 0 0 0 1 0 1 0 0 1 0 0 1 1 1 0 0 0 0 0 [329] 1 0 1 0 1 1 1 0 1 0 1 0 0 1 0 0 1 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 1 0 1 0 0 1 0 0 1 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 1 0 0 1 1 1 1 1 1 1 [411] 1 1 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 1 1 0 0 1 0 1 1 1 0 1 0 1 1 1 0 0 1 0 0 1 0 1 1 0 0 0 1 1 0 0 1 1 1 0 1 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 0 1 1 1 1 0 0 [493] 1 1 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 1 1 1 1 0 0 1 0 0 0 0 0 1 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 1 0 1 0 1 1 1 0 0 0 1 0 [575] 0 0 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 1 0 0 0 1 0 0 0 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 1 1 0 0 0 1 0 0 0 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 [657] 1 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 0 1 1 1 1 1 1 0 0 0 0 1 1 0 1 1 0 0 0 0 1 0 1 1 0 1 1 1 0 0 0 [739] 1 0 0 0 0 0 1 0 1 0 1 0 0 1 1 1 0 0 0 0 1 1 0 1 1 0 0 1 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1

Decoding requires adding the structure that you stripped

The key decoding step is that you lose byte structure after your encode() function converts from raw to integer. Let's look at an example with "A" (65 in ASCII or \u0041). You can see that as.integer() gives us a vector of bits:

"A" |>
    charToRaw() |>  # 41
    rawToBits() |> # 01 00 00 00 00 00 01 00
    as.integer() # 1 0 0 0 0 0 1 0

Each of the eight bits that made up one byte are flattened into separate integers, so you can't reconstruct the original raw value without re-grouping them into bytes. You can use packBits() after as.raw() to get these back:

decode <- function(message) {
    message |>
        rev() |>
        as.raw() |>
        packBits(type = "raw") |>
        rawToChar()
}

This allows us to decode() the message by reversing the order of encoding operations:

decode(encoded)
# [1] "café naïve — ∑ ∆ ∫ √ π λ 東京 😀 नमस्ते ✔ שלום ♥ ♦ ♣ ♠"

decode(encoded) == message
# [1] TRUE

You can always map a str to unicode using the latin-1 codec. Once you have a unicode, you can always encode it in utf-8:

data.decode('latin-1').encode("utf-8")