There are two easy and safe rules which work not only in sh, but also Bash.
1. Put the whole string in single quotes
This works for all chars except single quote itself. To escape the single quote, close the quoting before it, insert the single quote, and reopen the quoting.
'I'\''m a s@fe $tring which ends in newline
'
sed command: sed -e "s/'/'\\\\''/g; 1s/^/'/; \
/'/"
2. Escape every char with a backslash
This works for all characters except newline. For newline characters use single or double quotes. Empty strings must still be handled - replace with ""
\I\'\m\ \a\ \s\@\f\e\ \$\t\r\i\n\g\ \w\h\i\c\h\ \e\n\d\s\ \i\n\ \n\e\w\l\i\n\e"
"
sed command: sed -e 's/./\\&/g; 1{$s/^$/""/}; 1!s/^/"/; .
/"/'
2b. More readable version of 2
There's an easy safe set of characters, like [a-zA-Z0-9,._+:@%/-], which can be left unescaped to keep it more readable
I\'m\ a\ s@fe\ \$tring\ which\ ends\ in\ newline"
"
sed command: LC_ALL=C sed -e 's/[^a-zA-Z0-9,._+@%/-]/\\&/g; 1{$s/^$/""/}; 1!s/^/"/; .
/"/'
Note that in a sed program, one can't know whether the last line of input ends with a newline byte (except when it's empty). That's why both above sed commands assume it does not. You can add a quoted newline manually.
Note that shell variables are only defined for text in the POSIX sense. Processing binary data is not defined. For the implementations that matter, binary works with the exception of NUL bytes (because variables are implemented with C strings, and meant to be used as C strings, namely program arguments), but you should switch to a "binary" locale such as latin1.
(You can easily validate the rules by reading the POSIX spec for sh. For Bash, check the reference manual linked by @AustinPhillips.)
linux shell escape - execute commands without letters or backslashes
bash - Escaping a linux command - Unix & Linux Stack Exchange
bash - What characters are required to be escaped in command line arguments? - Unix & Linux Stack Exchange
shell - How do I exit or cancel a bad bash command? - Unix & Linux Stack Exchange
Videos
There are two easy and safe rules which work not only in sh, but also Bash.
1. Put the whole string in single quotes
This works for all chars except single quote itself. To escape the single quote, close the quoting before it, insert the single quote, and reopen the quoting.
'I'\''m a s@fe $tring which ends in newline
'
sed command: sed -e "s/'/'\\\\''/g; 1s/^/'/; \/'/"
2. Escape every char with a backslash
This works for all characters except newline. For newline characters use single or double quotes. Empty strings must still be handled - replace with ""
\I\'\m\ \a\ \s\@\f\e\ \$\t\r\i\n\g\ \w\h\i\c\h\ \e\n\d\s\ \i\n\ \n\e\w\l\i\n\e"
"
sed command: sed -e 's/./\\&/g; 1{$s/^$/""/}; 1!s/^/"/; ./"/'
2b. More readable version of 2
There's an easy safe set of characters, like [a-zA-Z0-9,._+:@%/-], which can be left unescaped to keep it more readable
I\'m\ a\ s@fe\ \$tring\ which\ ends\ in\ newline"
"
sed command: LC_ALL=C sed -e 's/[^a-zA-Z0-9,._+@%/-]/\\&/g; 1{$s/^$/""/}; 1!s/^/"/; ./"/'
Note that in a sed program, one can't know whether the last line of input ends with a newline byte (except when it's empty). That's why both above sed commands assume it does not. You can add a quoted newline manually.
Note that shell variables are only defined for text in the POSIX sense. Processing binary data is not defined. For the implementations that matter, binary works with the exception of NUL bytes (because variables are implemented with C strings, and meant to be used as C strings, namely program arguments), but you should switch to a "binary" locale such as latin1.
(You can easily validate the rules by reading the POSIX spec for sh. For Bash, check the reference manual linked by @AustinPhillips.)
Format that can be reused as shell input
Edit February 2021: Bash ${var@Q}
Under Bash, you could store your variable content with Parameter Expansion's @ command for Parameter transformation:
${parameter@operator} Parameter transformation. The expansion is either a > transforma‐ tion of the value of parameter or information about parameter itself, depending on the value of operator. Each operator is a single letter: Q The expansion is a string that is the value of parameter quoted in a format that can be reused as input. ... A The expansion is a string in the form of an assignment statement or declare command that, if evaluated, will recreate parameter with its attributes and value.
Sample:
'Hello\nGood world.\n'
$ echo "$var"
Hello
Good world.
$ echo "
'Hello\nGood world.\n'
$ echo "${var@A}"
var=$'Hello\nGood world.\n'
Old answer
There is a special printf format directive (%q) built for this kind of request:
printf [-v var] format [arguments] %q causes printf to output the corresponding argument in a format that can be reused as shell input.
Some samples:
read foo
Hello world
printf "%q\n" "$foo"
Hello\ world
printf "%q\n" $'Hello world!\n'
$'Hello world!\n'
This could be used through variables too:
printf -v var "%q" "$foo
"
echo "
'Hello world\n'
Quick check with all (128) ASCII bytes:
Note that all bytes from 128 to 255 have to be escaped.
This little loop will print all characters from 0x00 to 0x7f,
by using both: printf %q and ${var@Q} method.
for i in {0..127}; do
printf -v var %02X $i
printf -v var %b \\x$var
sign=E
printf -v res %q "$var"
[[ $var == "$res" ]] && sign=-
printf '%02X %s %-*s %-*s\n' $i $sign $(( 31 < i && i < 96 ? 2 : 8
)) "${res}" $(( 31 < i && i < 96 ? 3 : 8)) "${var@Q}"
done |
pr -w100 --sep-string='' -t4 |
sed 's/\o11\+/\o11/g'
This should render something like:
00 E '' '' 20 E \ ' ' 40 - @ '@' 60 E \` '`'
01 E 
'\001' 21 E \! '!' 41 - A 'A' 61 - a 'a'
02 E 
'\002' 22 E \" '"' 42 - B 'B' 62 - b 'b'
03 E 
'\003' 23 E \# '#' 43 - C 'C' 63 - c 'c'
04 E 
'\004' 24 E \
' 44 - D 'D' 64 - d 'd'
05 E 
'\005' 25 - % '%' 45 - E 'E' 65 - e 'e'
06 E 
'\006' 26 E \& '&' 46 - F 'F' 66 - f 'f'
07 E 
'\a' 27 E \' \' 47 - G 'G' 67 - g 'g'
08 E 
'\b' 28 E \( '(' 48 - H 'H' 68 - h 'h'
09 E 
'\t' 29 E \) ')' 49 - I 'I' 69 - i 'i'
0A E 
'\n' 2A E \* '*' 4A - J 'J' 6A - j 'j'
0B E $'\v' $'\v' 2B - + '+' 4B - K 'K' 6B - k 'k'
0C E $'\f' $'\f' 2C E \, ',' 4C - L 'L' 6C - l 'l'
0D E 
'\r' 2D - - '-' 4D - M 'M' 6D - m 'm'
0E E 
'\016' 2E - . '.' 4E - N 'N' 6E - n 'n'
0F E 
'\017' 2F - / '/' 4F - O 'O' 6F - o 'o'
10 E 
'\020' 30 - 0 '0' 50 - P 'P' 70 - p 'p'
11 E 
'\021' 31 - 1 '1' 51 - Q 'Q' 71 - q 'q'
12 E 
'\022' 32 - 2 '2' 52 - R 'R' 72 - r 'r'
13 E 
'\023' 33 - 3 '3' 53 - S 'S' 73 - s 's'
14 E 
'\024' 34 - 4 '4' 54 - T 'T' 74 - t 't'
15 E 
'\025' 35 - 5 '5' 55 - U 'U' 75 - u 'u'
16 E 
'\026' 36 - 6 '6' 56 - V 'V' 76 - v 'v'
17 E 
'\027' 37 - 7 '7' 57 - W 'W' 77 - w 'w'
18 E 
'\030' 38 - 8 '8' 58 - X 'X' 78 - x 'x'
19 E 
'\031' 39 - 9 '9' 59 - Y 'Y' 79 - y 'y'
1A E 
'\032' 3A - : ':' 5A - Z 'Z' 7A - z 'z'
1B E 
'\E' 3B E \; ';' 5B E \[ '[' 7B E \{ '{'
1C E 
'\034' 3C E \< '<' 5C E \\ '\' 7C E \| '|'
1D E 
'\035' 3D - = '=' 5D E \] ']' 7D E \} '}'
1E E 
'\036' 3E E \> '>' 5E E \^ '^' 7E E \~ '~'
1F E 
'\037' 3F E \? '?' 5F - _ '_' 7F E $'\177' $'\177'
Where
- first field is hexadecimal value of byte,
- second contain
Eif character need to be escaped, - third field show escaped presentation of character and
- last field show useable version printed by
${var@Q}syntax.
Small function looking for limited bunch of characters
For fun, here is another way for looping over a string, grouping all characters by the need to be escaped.
specialCharsFromString() {
local {q,}char bunch{_0,_1} \
special="${1:-'\`\"/\!@#\$%^&*()-_+={\}[]|;:,.<>? '}"
while IFS= LANG=C LC_ALL=C read -d '' -rn 1 char; do
printf -v qchar %q "$char"
[[ $char == "$qchar" ]]
local -n bunch=bunch_$?
bunch+=(${char@Q})
done < <(printf %s "$special");
printf 'Characters who %sneed to be escaped:\n%s\n' \
"doesn't " "${bunch_0[*]}" "" "${bunch_1[*]}"
}
specialCharsFromString $'`!@#$%^&*()-_+={}|[]\\;\':",.<>?/ '
Characters who doesn't need to be escaped:
'@' '%' '-' '_' '+' '=' ':' '.' '/'
Characters who need to be escaped:
'`' '!' '#' '$' '^' '&' '*' '(' ')' '{' '}' '|' '[' ']' '\' ';' \' '"' ',' '<' '>' '?' ' '
Why ,?
You could see some characters that don't always need to be escaped, like ,, } and {.
So not always, but sometimes:
echo test 1, 2, 3 and 4,5.
test 1, 2, 3 and 4,5.
or
echo test { 1, 2, 3 }
test { 1, 2, 3 }
but care:
echo test{1,2,3}
test1 test2 test3
echo test\ {1,2,3}
test 1 test 2 test 3
echo test\ {\ 1,\ 2,\ 3\ }
test 1 test 2 test 3
echo test\ {\ 1\,\ 2,\ 3\ }
test 1, 2 test 3
See Brace Expansion chapter in Bash's man page:
man -P'less +/Brace\ Expansion' bash
Note about percent sign %.
No, percent sign don't need to be escaped in any POSIX shell compatible!
But in cron's crontab!! This is a common issue as syntax used in crontab is mostly POSIX shell compatible. but man -Pless\ +/% 5 crontab:
... Percent-signs (%) in the command, unless escaped with backslash (), will be changed into newline characters...
If you try to use timestamp is crontab, you have to escape %:
* * * * * echo one more line >>file-$(date +\%F).log
Note about dashes sign -.
There isn't any reason to escape a dash in a string except if you try to use them at begin of string as the first argument of a regular command:
printf '- %3d %s\n' $((count++)) "Some String"
bash: printf: - : invalid option
ls -dashedFilename
ls: invalid option -- 'e'
Even escaped, this won’t go better:
ls \-dashedFilename
ls: invalid option -- 'e'
For this, the recommended way is to use double dash --:
From POSIX.1-2017 Utility Conventions Guideline 10:
Guideline 10:
The first
--argument that is not an option-argument should be accepted as a delimiter indicating the end of options. Any following arguments should be treated as operands, even if they begin with the-character.
printf -- '- %3d %s\n' $((count++)) "Some String"
- 2 Some String
ls -- -dashedFilename
ls: cannot access '-dashedFilename': No such file or directory
Ok, they don't exist.
Alternative: Simply avoid dashes at begin of any string:
printf '\55 %3d %s\n' $((count++)) "Some other String"
- 3 Some other String
printf '\x2d %3d %s\n' $((count++)) "Some other String"
- 4 Some other String
Using any of octal or hexadecimal representation of dash
ls ./-dashedFilename
ls: cannot access './-dashedFilename': No such file or directory
Using relative or full path.
Let's say I want to execute the command:
ls
But I can't use letters, so one way would be:
$'\154'$'\163'
But I can't use backslashes as well.
Now what?
# & * ( ) blank tab | ' ; " < > \ ~ ` [ ] { } Presumably, that vzctl exec works like eval. That is, it constructs a shell command line from the concatenation of the arguments it receives. So you should pass it a valid shell command line as argument which would mean another level of quoting.
vzctl exec VZID '
sed -i '\''s/\/>/address="$IP\/255.255.255.0"\/\>/'\'' "file.xml"'
Note that $IP will not be expanded. For that, you'd need either:
vzctl exec VZID '
sed -i '\''s/\/>/address="'"$IP"'\/255.255.255.0"\/\>/'\'' "file.xml"'
For it to be expanded by the shell that interprets that vzctl command or:
vzctl exec VZID '
sed -i '\''s/\/>/address="'\''"$IP"'\''\/255.255.255.0"\/\>/'\'' "file.xml"'
For it to be expanded by the shell that vzctl exec invokes (assuming that shell has access to such a $IP variable).
Note that it may be easier to use a here-document and store that code in a variable for clarity like:
code=$(cat <<EOF
sed -i 's|/>|address="$IP/255.255.255.0"/>|' file.xml
EOF
)
(that form of here document does expand the variables (here $ID)).
And run it as
vzctl exec VZID "$code"
Have you tried to wrap whole command in quotes and espace the internal with backslashes?
I mean:
vzctl exec VZID 'sed -i \'s/\/>/address=\"$IP\/255.255.255.0\"\/\>/\' \"file.xml\"'
I'd check also double quotes if the single don't work, but I think they should.
The following characters have special meaning to the shell itself in some contexts and may need to be escaped in arguments:
| Character | Unicode | Name | Usage |
|---|---|---|---|
` |
U+0060 (Grave Accent) | Backtick | Command substitution |
~ |
U+007E | Tilde | Tilde expansion |
! |
U+0021 | Exclamation mark | History expansion |
# |
U+0023 Number sign | Hash | Comments |
$ |
U+0024 | Dollar sign | Parameter expansion |
& |
U+0026 | Ampersand | Background commands |
* |
U+002A | Asterisk | Filename expansion and globbing |
( |
U+0028 | Left Parenthesis | Subshells |
) |
U+0029 | Right Parenthesis | Subshells |
|
U+0009 | Tab (⇥) |
Word splitting (whitespace) |
{ |
U+007B Left Curly Bracket | Left brace | Brace expansion |
[ |
U+005B | Left Square Bracket | Filename expansion and globbing |
| |
U+007C Vertical Line | Vertical bar | Pipelines |
\ |
U+005C Reverse Solidus | Backslash | Escape character |
; |
U+003B | Semicolon | Separating commands |
' |
U+0027 Apostrophe | Single quote | String quoting |
" |
U+0022 Quotation Mark | Double quote | String quoting with interpolation |
↩ |
U+000A Line Feed | Newline | Line break |
< |
U+003C | Less than | Input redirection |
> |
U+003E | Greater than | Output redirection |
? |
U+003F | Question mark | Filename expansion and globbing |
|
U+0020 | Space | Word splitting1 (whitespace) |
Some of those characters are used for more things and in more places than the one I linked.
There are a few corner cases that are explicitly optional:
!can be disabled withset +H, which is the default in non-interactive shells.{can be disabled withset +B.*and?can be disabled withset -forset -o noglob.=Equals sign (U+003D) also needs to be escaped ifset -korset -o keywordis enabled.
Escaping a newline requires quoting — backslashes won't do the job. Any other characters listed in IFS will need similar handling. You don't need to escape ] or }, but you do need to escape ) because it's an operator.
Some of these characters have tighter limits on when they truly need escaping than others. For example, a#b is ok, but a #b is a comment, while > would need escaping in both contexts. It doesn't hurt to escape them all conservatively anyway, and it's easier than remembering the fine distinctions.
If your command name itself is a shell keyword (if, for, do) then you'll need to escape or quote it too. The only interesting one of those is in, because it's not obvious that it's always a keyword. You don't need to do that for keywords used in arguments, only when you've (foolishly!) named a command after one of them. Shell operators ((, &, etc) always need quoting wherever they are.
1Stéphane has noted that any other single-byte blank character from your locale also needs escaping. In most common, sensible locales, at least those based on C or UTF-8, it's only the whitespace characters above. In some ISO-8859-1 locales, U+00A0 no-break space is considered blank, including Solaris, the BSDs, and OS X (I think incorrectly). If you're dealing with an arbitrary unknown locale, it could include just about anything, including letters, so good luck.
Conceivably, a single byte considered blank could appear within a multi-byte character that wasn't blank, and you'd have no way to escape that other than putting the whole thing in quotes. This isn't a theoretical concern: in an ISO-8859-1 locale from above, that A0 byte which is considered a blank can appear within multibyte characters like UTF-8 encoded "à" (C3 A0). To handle those characters safely you would need to quote them "à". This behaviour depends on the locale configuration in the environment running the script, not the one where you wrote it.
I think this behaviour is broken multiple ways, but we have to play the hand we're dealt. If you're working with any non-self-synchronising multibyte character set, the safest thing would be to quote everything. If you're in UTF-8 or C, you're safe (for the moment).
In GNU Parallel this is tested and used extensively:
$a =~ s/[\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\<\=\>\~\|\; \"\!\$\&\'\202-\377]/\\$&/go;
# quote newline as '\n'
$a =~ s/[\n]/'\n'/go;
It is tested in bash,dash,ash,ksh,zsh, and fish. Some of the characters do not need quoting in some (versions) of the shells, but the above works in all tested shells.
If you simply want a string quoted, you can pipe it into parallel --shellquote:
printf "&*\t*!" | parallel --shellquote
You can always try the obvious things like ^C, ^D (eof), Escape etc., but if all fails I usually end up suspending the command with ^Z (Control-Z) which puts me back into the shell.
I then do a ps command and note the PID (process id) of the command and then issue a kill thePID (kill -9 thePID if the former didn't work) command to terminate the application.
Note that this is not a tidy (no pun intended) way to terminate the application/command and you run the risk of perhaps no saving some data etc.
An example (I'd have used tidy but I don't have it installed):
$ gnuplot
G N U P L O T
Version 4.2 patchlevel 6
....
Send bug reports and suggestions to <http://sourceforge.net/projects/gnuplot>
Terminal type set to 'wxt'
gnuplot>
gnuplot> ##### typed ^Z here
[1]+ Stopped gnuplot
$ ps
PID TTY TIME CMD
1681 pts/1 00:00:00 tcsh
1690 pts/1 00:00:00 bash
1708 pts/1 00:00:00 gnuplot
1709 pts/1 00:00:00 ps
$ kill 1708 ###### didn't kill the command as ps shows
$ ps
PID TTY TIME CMD
1681 pts/1 00:00:00 tcsh
1690 pts/1 00:00:00 bash
1708 pts/1 00:00:00 gnuplot
1710 pts/1 00:00:00 ps
$ kill -9 1708 ### -9 did the trick
$
[1]+ Killed gnuplot
$ ps
PID TTY TIME CMD
1681 pts/1 00:00:00 tcsh
1690 pts/1 00:00:00 bash
1711 pts/1 00:00:00 ps
Try pressing Ctrl-D or Ctrl-C. If it fails, kill the process .
Trying with the tidy command you mentioned, Ctrl-D works.
This is happening because \ is an escape character.
Either use quoting
path/to/forticlientsslvpn_cli --server <host>:<port> --vpnuser 'testpass\101'
or use an escaped backslash:
path/to/forticlientsslvpn_cli --server <host>:<port> --vpnuser testpass\\101
For completeness, with the bash shell and on ASCII based systems, with quoting alone:
'testpass\101'(by far the best)testpass\\101"testpass\101"or"testpass\\101"(latter better)$'testpass\\101'$'testpass\u005c101'or$'testpass\U0000005c101'(U+005C being the Unicode code point for backslash)$'testpass\x5c101'(where 0x5C is the byte value of the ASCII encoding of\)$'testpass\134101'(same in octal)
For more details, see How to use a special character as a normal one?