In what format is the response you get? There is no such thing as a text file. There are only binary files. HTTP is also 100% binary.

Text is the interpretation of bytes, and it only exists as part of running application. You can never, ever write text to a file. You can only convert the text to bytes (using various ways) and write the bytes.

Therefore, ask yourself why the bytes you received cannot be interpreted by notepad.exe as text. Maybe the response is not directly text but a ZIP file or something.

1. You can guess the format with a hex editor
2. You can ask the website owner

In the Edit menu of Notepad++ (I'm using v5.9.2) under Paste Special there are actions to Copy, Cut, and Paste Binary Content.

Notepad++ is an editor for text files, which are files that end with special binary characters that denote an end-of-line. It will always think that your text-line is found between two end-of-line characters, and will mostly ignore the end-of-line itself in its operations.

This has the potential to destroy the file when converting to Base64, since Notepad++ will convert the text between the end-of-lines, but may cause a problem with not correctly handling the end-of-lines.

In fact, it's pretty amazing for me that this has worked correctly for you for some files, and I can readily understand that it didn't for others.

I think you should really look for another solution than Notepad++ for converting to Base64.

Your question really has two parts: (1) how do I identify an unknown encoding and (2) how do I convert that to something useful?

The first part is the real challenge, and really cannot be answered in universal terms -- in the general case, there is no reliable way to identify an unknown 8-bit encoding. Some encodings give you good hints (UTF-8 is an excellent example) and in many cases, if you have a good idea what the text is supposed to represent, the problem can be solved.

A mapping of 8-bit character meanings can be helpful (cough, the link is to mine) and in this case quickly hints at Windows code page 1251. Kudos for the hex dumps and the picture with the representation you expect!

With that out of the way, converting is easy.

iconv -f cp1251 -t utf-8 non_ascii.txt >utf8.txt

Provided your Linux system is set up to use UTF-8 at the terminal, your grep command should work on utf-8.txt now.

The indication that some of the text is "ANSI" (which is a bogus term anyway) is probably just a red herring -- as far as I can tell, everything in your excerpt looks like well-formed CP1251.

Some tools like chardet do a reasonable job of at least steering you in the right direction, though you have to understand that, like a human expert, they have to guess what the text is supposed to represent. There are corner cases where they just don't have enough information to guess correctly, either because there are several candidate encodings with very few differences (for example, Latin-1 vs Latin-9 vs Windows-1252, all of which also overlap with plain 7-bit US-ASCII in the first 128 positions) or because the input doesn't contain enough information to establish any common patterns.

od -An -vtx1 Check.tar > Check.txt

You need -v or od will condense sequences of identical bytes.

For the reverse:

LC_ALL=C tr -cd 0-9a-fA-F < Check.txt | xxd -r -p > Check.tar

Or:

perl -ape '$_=pack "(H2)*", @F' Check.txt > Check.tar

If your purpose is to transfer files over a channel that only supports ASCII text, then there are dedicated tools for that like uuencode:

tar cf - myfiles.* | xz | uuencode myfiles.tar.xz | that-channel 

And to recover those files on the other end:

uudecode < file.uu

would recreate myfiles.tar.xz.

Or:

uudecode -o - < file.uu | xz -d | tar xf -

To extract the files.

In SynWrite editor this converting of NULL char can be done. Synwrite has text-converters (Run menu): described in help file topic.

PSPad has similar text-conv feature (Tools menu).

Or you can use a regex to replace [\x00-\x19] with new string.