Well, I'm using NotePad++ and I can't see that at all! What is the best text file reader for this kind of problems?
The problem is, a ‘good’ text editor should be able to load all text encodings transparently — even stupid broken ones like UTF-8-plus-BOM — which would prevent you from seeing the problem. Sure, a good text editor should save UTF-8 without the bogus-BOM, or at least give you the option to do so, but you won't know to re-save it if you don't see the faux-BOM there.
The reason you see the three high-bytes at the start of the file in TextMate is actually because TextMate has got it wrong and guessed the encoding as Latin-1 instead of UTF-8. This presumably reproduces the behaviour of the service you're sending to which don't know about Unicode, but it's not really a desirable feature in itself. It's also why the æs and øs haven't come out.
If you want to see every byte in the file explicitly, what you want isn't really a text editor, but a hex editor. There are lots to choose from, eg. xvi32 on Windows.
And then fix your application to not produce bogus BOMs; they have no place in a UTF-8 file anyway, never mind the problems it causes to non-Unicode applications. [I don't know what the application is written in, but a common cause of unwanted BOMs is using .NET's Encoding.UTF8 encoding. A new UTF8Encoding(false) would be preferable.]
Whether the service you're sending to wants UTF-8 or some other encoding is in any case something you'll have to ask the operators of that service. If they're already describing the high-bytes for æ et al in your file as inherently ‘invalid’, you may be facing a situation where they don't support any non-ASCII characters at all, in which case you'll have to consider transliterating characters appropriately for the target language, eg. æ->ae.
Well, I'm using NotePad++ and I can't see that at all! What is the best text file reader for this kind of problems?
The problem is, a ‘good’ text editor should be able to load all text encodings transparently — even stupid broken ones like UTF-8-plus-BOM — which would prevent you from seeing the problem. Sure, a good text editor should save UTF-8 without the bogus-BOM, or at least give you the option to do so, but you won't know to re-save it if you don't see the faux-BOM there.
The reason you see the three high-bytes at the start of the file in TextMate is actually because TextMate has got it wrong and guessed the encoding as Latin-1 instead of UTF-8. This presumably reproduces the behaviour of the service you're sending to which don't know about Unicode, but it's not really a desirable feature in itself. It's also why the æs and øs haven't come out.
If you want to see every byte in the file explicitly, what you want isn't really a text editor, but a hex editor. There are lots to choose from, eg. xvi32 on Windows.
And then fix your application to not produce bogus BOMs; they have no place in a UTF-8 file anyway, never mind the problems it causes to non-Unicode applications. [I don't know what the application is written in, but a common cause of unwanted BOMs is using .NET's Encoding.UTF8 encoding. A new UTF8Encoding(false) would be preferable.]
Whether the service you're sending to wants UTF-8 or some other encoding is in any case something you'll have to ask the operators of that service. If they're already describing the high-bytes for æ et al in your file as inherently ‘invalid’, you may be facing a situation where they don't support any non-ASCII characters at all, in which case you'll have to consider transliterating characters appropriately for the target language, eg. æ->ae.
An easy way to view this kind of stuff in Windows is to use the "type" command.
I would do something like this:
type filename.txt | more
bash - Identify and remove specific hidden characters from text file - Stack Overflow
Hidden character in text file
Likely an ASCII character that did not convert correctly. Try using NotePad++ or something non-Microsoft to make sure the CLRF (carriage return, line feed) characters are displaying as expected. Otherwise it could be a control character that is being used for some reason.
More on reddit.combinary - How to find non-printable character in text file - Stack Overflow
Hidden characters in text file
How do I detect invisible characters in my text?
Where do invisible characters come from?
What are invisible characters?
I'm in the process of reverse engineering some of my company's existing scripts, and after dumping the output into OneNote, I get a little boxed "?" in some of the spaces, indicating a special character.
Now...in notepad this character is nowhere to be seen, however, I do get a little phantom character when scrolling through the text
i.e. To get to the end of
...
I need to scroll over one of the periods twice, as the cursor does not move the first time you press left/right. Now bear in mind, that this is in notepad, so I didn't think I'd see anything weird in the character department.
It might be worth noting that this is a .sed file. The old sys admin guy installed the .exe ports of all the common Unix commands onto all of our Windows servers and is running this as a cron job..on Windows. Please don't ask.
Any insight would be very much appreciated.
UPDATE#
The hidden character is called a File Separator and is a non visible character used for functions rather than text.
Apparently the script it appears in is what's called a pipe separated value to comma separated value script. It's contents are as follows:
/--- ---/d s/ */ /g s/,,/,/g s/^ *// s/"/'/g s/.*/"&/ s/$/"/ s/ /","/g s/" */"/g s/ *"/"/g
If you copy the text the FS hidden character should be in there too. After some research, I found that this is pretty common when taking SQL data and transforming it into raw data, as these Reg Exes strip away any odd spaces, take out any SQL formatting, leaving you with raw text.
So, for the sed command:
| sed -f psv2csv.sed > Outputfile.csv
Where we pipe data in from the SQLCMD, grep the pertinent lines and THEN Sed. The -f specifies that a input script is used for the RegEx.
Maybe you'll find it useful!
Likely an ASCII character that did not convert correctly. Try using NotePad++ or something non-Microsoft to make sure the CLRF (carriage return, line feed) characters are displaying as expected. Otherwise it could be a control character that is being used for some reason.
It's contents are as follows:
This is the ugliest goddamn thing I've seen in my life. And I'm sure it mangles data on edge cases.