tl;dr

Inside "...":

• $(...) is only needed to embed entire expressions or commands (e.g. "PSVersionTable.PSVersion)")

• ${...} is only needed if a stand-alone reference to a variable with a non-special name needs delimiting (disambiguation), so that subsequent characters in the string aren't interpreted as part of the variable name (e.g, "${foo}_bar" correctly embeds variable $foo; without {...}, PowerShell would look for variable $foo_bar; notably, a subsequent : requires this technique too: "${foo}:bar"; by contrast, something like "$foo!" also works without {...})

Independently of use in "...", ${...} is also needed for variable names containing special characters (e.g. ${a-b} = 42)

For a comprehensive overview of PowerShell's expandable strings (string interpolation), see this answer.

Background information:

To complement marsze's helpful answer:

${...} (enclosing the variable name in { and }) is indeed always necessary if a variable name contains special characters, such as spaces, . or -.

• Not special are _ and - surprisingly and problematically - ?.

  • For the exact rules, see the relevant section of the conceptual "about_Variables" help topic, Variable Names that Include Special Characters.

• (The first occurrence of) : in a name is invariably interpreted as terminating either a PowerShell drive reference - in the context of namespace variable notation - or a scope specifier, irrespective of whether {...} enclosure is used or required (e.g., in $env:USERNAME or ${env:USERNAME}, env refers to the Env: PowerShell drive representing all environment variables; in $script:foo or ${script:foo}, script refers to the script's scope and its variables).

Note:

• ${...} - the syntax for disambiguating a variable name - is not to be confused with $(...), the subexpression operator, which is needed to embed any expression or command that goes beyond a stand-alone variable reference inside "...", an expandable (interpolating) string.

• As such, these two syntax forms are independent of one another and may need to be combined in a given situation; e.g. "$var" / "${var}" work fine, but "$var.someProperty" / "${var}.someProperty" do not: you need "var.someProperty)" / "{var}.someProperty)"

In the context of "...", there is another reason to use ${...}, even if the variable name itself doesn't need it:

If you need to delineate the variable name from directly following non-whitespace characters, notably including ::

$foo = 'bar'  # example variable

# INCORRECT: PowerShell assumes that the variable name is 'foobarian', not 'foo'
PS> "A $foobarian."
A .  # Variable $foobarian doesn't exist -> reference expanded to empty string.

# CORRECT: Use {...} to delineate the variable name:
PS> "A ${foo}barian."
A barbarian.

# INCORRECT: PowerShell assumes that 'foo:' is a *namespace* (drive) reference
#            (such as 'env:' in $env:PATH) and FAILS:
PS> "$foo: bar"
Variable reference is not valid. ':' was not followed by a valid variable name character. 
Consider using ${} to delimit the name.

# CORRECT: Use {...} to delineate the variable name:
PS> "${foo}: bar"
bar: bar

See this answer for a comprehensive overview of PowerShell string-expansion rules.

Note that you need the same technique when string expansion is implicitly applied, in the context of passing an unquoted argument to a command; e.g.:

# INCORRECT: The argument is treated as if it were enclosed in "...",
#            so the same rules apply.
Write-Output $foo:/bar

# CORRECT
Write-Output ${foo}:/bar

Finally, a somewhat obscure alternative is to `-escape the first character after the variable name, but the problem is that this only works as expected with characters that aren't part of escape sequences (see about_Special_Characters):

# OK: because `: is not an escape sequence.
PS> "$foo`: bar"
bar: bar

# NOT OK, because `b is the escape sequence for a backspace character.
PS> "$foo`bar"
baar # The `b "ate" the trailing 'r' of the variable value
     # and only "ar" was the literal part.

PowerShell only supports interpolation at the time of a variable creation, unless you are talking about calculated object properties:

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_calculated_properties?view=powershell-7

If you are looking for f-strings see here: https://ss64.com/ps/syntax-f-operator.html

tl;dr

Inside "...":

• $(...) is only needed to embed entire expressions or commands (e.g. "$($PSVersionTable.PSVersion)")

• ${...} is only needed if a stand-alone reference to a variable with a non-special name needs delimiting (disambiguation), so that subsequent characters in the string aren't interpreted as part of the variable name (e.g, "${foo}_bar" correctly embeds variable $foo; without {...}, PowerShell would look for variable $foo_bar; notably, a subsequent : requires this technique too: "${foo}:bar"; by contrast, something like "$foo!" also works without {...})

Independently of use in "...", ${...} is also needed for variable names containing special characters (e.g. ${a-b} = 42)

For a comprehensive overview of PowerShell's expandable strings (string interpolation), see this answer.

Background information:

To complement marsze's helpful answer:

${...} (enclosing the variable name in { and }) is indeed always necessary if a variable name contains special characters, such as spaces, . or -.

• Not special are _ and - surprisingly and problematically - ?.

  • For the exact rules, see the relevant section of the conceptual "about_Variables" help topic, Variable Names that Include Special Characters.

• (The first occurrence of) : in a name is invariably interpreted as terminating either a PowerShell drive reference - in the context of namespace variable notation - or a scope specifier, irrespective of whether {...} enclosure is used or required (e.g., in $env:USERNAME or ${env:USERNAME}, env refers to the Env: PowerShell drive representing all environment variables; in $script:foo or ${script:foo}, script refers to the script's scope and its variables).

Note:

• ${...} - the syntax for disambiguating a variable name - is not to be confused with $(...), the subexpression operator, which is needed to embed any expression or command that goes beyond a stand-alone variable reference inside "...", an expandable (interpolating) string.

• As such, these two syntax forms are independent of one another and may need to be combined in a given situation; e.g. "$var" / "${var}" work fine, but "$var.someProperty" / "${var}.someProperty" do not: you need "$($var.someProperty)" / "$(${var}.someProperty)"

In the context of "...", there is another reason to use ${...}, even if the variable name itself doesn't need it:

If you need to delineate the variable name from directly following non-whitespace characters, notably including ::

$foo = 'bar'  # example variable

# INCORRECT: PowerShell assumes that the variable name is 'foobarian', not 'foo'
PS> "A $foobarian."
A .  # Variable $foobarian doesn't exist -> reference expanded to empty string.

# CORRECT: Use {...} to delineate the variable name:
PS> "A ${foo}barian."
A barbarian.

# INCORRECT: PowerShell assumes that 'foo:' is a *namespace* (drive) reference
#            (such as 'env:' in $env:PATH) and FAILS:
PS> "$foo: bar"
Variable reference is not valid. ':' was not followed by a valid variable name character. 
Consider using ${} to delimit the name.

# CORRECT: Use {...} to delineate the variable name:
PS> "${foo}: bar"
bar: bar

See this answer for a comprehensive overview of PowerShell string-expansion rules.

Note that you need the same technique when string expansion is implicitly applied, in the context of passing an unquoted argument to a command; e.g.:

# INCORRECT: The argument is treated as if it were enclosed in "...",
#            so the same rules apply.
Write-Output $foo:/bar

# CORRECT
Write-Output ${foo}:/bar

Finally, a somewhat obscure alternative is to `-escape the first character after the variable name, but the problem is that this only works as expected with characters that aren't part of escape sequences (see about_Special_Characters):

# OK: because `: is not an escape sequence.
PS> "$foo`: bar"
bar: bar

# NOT OK, because `b is the escape sequence for a backspace character.
PS> "$foo`bar"
baar # The `b "ate" the trailing 'r' of the variable value
     # and only "ar" was the literal part.

Is this supported?

No, formatting is not supported during string expansion

As you might have noticed, string expansion in PowerShell works by naively resolving subexpressions nested in double-quoted strings - there are no {} placeholder constructs.

If you want string formatting, -f is the way to go.

FWIW, $s -f $a is directly translated to a String.Format($s, $a) call

For value types that support string formatting you can usually also call ToString() with a format string (just like in C#):

PS C:\> $a = 1 / 3 
PS C:\> $a.ToString("F2")
0.33