I don't know how you are trying to process your text, but apparently you are trying to run tr, which gives you the error message tr: Illegal byte sequence. This happens when its input is not a byte sequence that corresponds to a valid UTF-8 encoding (not all byte sequences correspond to the UTF-8 encoding of a series of Unicode characters).

I do not know what kind of file you are trying to process, but in a MacOS X environment the command file -I might give you an idea of the encoding that is actually there.

If it is just a matter of recoding your file, then iconv is a useful program. You can use it to recode to UTF-8 encoding by using iconv -f ... -t utf8 (where ... is the encoding of your original file, run iconv -l for a list of encodings that are available that way).

Or if you really want to remove the special characters in your file (as you state in the title of your question), you can use iconv -f ... -t ascii//TRANSLIT. In this last case, the "special characters" will be approximated by normal ASCII characters.

You can use sed:

sed -e 's/[;,()'\'']/ /g;s/  */ /g' input.sql > output.txt

or, if you rather want to specify what characters to keep:

sed -e 's/[^a-zA-Z*0-9]/ /g;s/  */ /g' input.sql > output.txt

You can use the replace() method to to replace the \r\n substrings with an empty string.

with open(data, 'r', encoding='utf-8') as myfile:
    with open('out.txt', 'a',  encoding='utf-8') as output:
        for line in myfile:
            output.write(line.replace('\r\n', '').rstrip())

This is certainly much, much longer than it needs to be. All you need is the tr utility, plus a loop and redirections to act on the files that are passed as arguments to the script.

#!/bin/sh
for file do
  tr -d '\r\t' <"$file" >"$file.safe"
done

With the option -d, tr removes the specified characters. The characters to remove are passed together as the first non-option argument. You can use backslash escapes to represent special characters: \n for a newline (^J), \r for a carriage return (^M), \t for a tab (^I), etc.

I haven't reproduced the code for asking the user because it's pointless. Directories will cause an error with redirection anyway, and it's really the job of the caller not to request a nonsensical action such as treating a directory as a regular file, so I also skipped that part.

If you want to replace the original file, write to a temporary file then move the result in place.

#!/bin/sh
for file do
  tmp="$(TMPDIR=$(dirname -- "$file") mktemp)"
  tr -d '\r\t' <"$file" >"$tmp" && mv -f -- "$tmp" "$file"
done

The temporary file name is constructed using mktemp so that the script is robust. It will work as long as you have write permission to the directory containing the file, without risking overwriting an existing file. It's secure even if that directory is writable by other users who might try to inject other data (a potential problem in /tmp).

The mv command is only invoked if the call to tr succeeded, so there's no risk of losing data if tr fails, e.g. because the disk becomes full midway through.

If you want to avoid replacing the file by a new, identical file if it doesn't contain any special characters, there are two ways:

• You can check for the special characters first. There are several ways to do it. One way is to remove everything except those special characters and count the number of resulting characters. As an optimization, pipe through head -c 1 so that you don't need to go through the whole file if a special character is found close to the top: that way the count is 0 if there's nothing to do and 1 otherwise.

  if [ "$(tr -dc '\r\t' <"$file" | head -c 1 | wc -c)" -ne 0 ]; then
    tr -d '\r\t' <"$file" >"$tmp" && mv -f -- "$tmp" "$file"
  fi
  

• You can do the transformation, then check if it's identical to the original. This can be slower if the files are often already in the desired state. On the other hand, this technique generalizes to cases where it isn't easy to determine whether the file is in the desired state.

  tr -d '\r\t' <"$file" >"$tmp" &&
  if cmp -s "$tmp" "$file"; then
    rm -- "$tmp"
  else
    mv -f -- "file"
  fi
  

Remove everything except the printable characters (character class [:print:]), with sed:

sed $'s/[^[:print:]\t]//g' file.txt

[:print:] includes:

• [:alnum:] (alpha-numerics)
• [:punct:] (punctuations)
• space

The ANSI C quoting ($'') is used for interpreting \t as literal tab inside $'' (in bash and alike).

Use sed:

sed -i '1s/^\[//' file

or if your version of sed does not have -i:

sed '1s/^\[//' file > file.tmp && mv file.tmp file

Explanation:

• -i Edit file in place (Alternative: write output to .tmp file and move back to original name).
• 1 in first line, do the following:
• s/pattern/replacement/modifiers substitute pattern with replacement and use the given modifiers.
  In your case: pattern is ^\[ for match ] in the very beginning of the line ^ with empty replacement and no modifiers.

import re
string = open('a.txt').read()
new_str = re.sub('[^a-zA-Z0-9\n\.]', ' ', string)
open('b.txt', 'w').write(new_str)

It will change every non alphanumeric char to white space.

Remove characters that are not within the ascii table (11,12,40-176)

\11 = tab

\12 = new line

\40-176 = ( to ~ this range includes all letters and symbols present in the keyboard

cat test.txt | tr -cd '\11\12\40-\176' > temp && mv temp test.txt

NOTE: If your data has special characters that are not in the ascii table, they might be removed as well

Here's what worked:

cat DetectX_Search.txt | tr '\r' '\n' > DetectX.txt

MrMuscle:bin mnewman$ cat DetectX.txt
Search time: Sunday, April 28, 2019 at 10:40:35
Search complete. Nothing found.

Python can do the job. The process here is simple, we read in all the lines into list while simultaneously replacing the UTF escape character ( which is \u001b ), and then print out lines again, but without the escape character. The < input.txt sends old text to python command , and > new_file.txt sends text to new file.

Script:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import sys
lines=[l.strip().replace(u"\u001b","") for l in sys.stdin]
print("\n".join(lines))

Save it as delete_escape.py, make it executable with chmod +x ./delete_escape.py, and call it like so:

./delete_escape.py < input.txt > output.txt

Results: