tl;dr

Inside "...":

• $(...) is only needed to embed entire expressions or commands (e.g. "PSVersionTable.PSVersion)")

• ${...} is only needed if a stand-alone reference to a variable with a non-special name needs delimiting (disambiguation), so that subsequent characters in the string aren't interpreted as part of the variable name (e.g, "${foo}_bar" correctly embeds variable $foo; without {...}, PowerShell would look for variable $foo_bar; notably, a subsequent : requires this technique too: "${foo}:bar"; by contrast, something like "$foo!" also works without {...})

Independently of use in "...", ${...} is also needed for variable names containing special characters (e.g. ${a-b} = 42)

For a comprehensive overview of PowerShell's expandable strings (string interpolation), see this answer.

Background information:

To complement marsze's helpful answer:

${...} (enclosing the variable name in { and }) is indeed always necessary if a variable name contains special characters, such as spaces, . or -.

• Not special are _ and - surprisingly and problematically - ?.

  • For the exact rules, see the relevant section of the conceptual "about_Variables" help topic, Variable Names that Include Special Characters.

• (The first occurrence of) : in a name is invariably interpreted as terminating either a PowerShell drive reference - in the context of namespace variable notation - or a scope specifier, irrespective of whether {...} enclosure is used or required (e.g., in $env:USERNAME or ${env:USERNAME}, env refers to the Env: PowerShell drive representing all environment variables; in $script:foo or ${script:foo}, script refers to the script's scope and its variables).

Note:

• ${...} - the syntax for disambiguating a variable name - is not to be confused with $(...), the subexpression operator, which is needed to embed any expression or command that goes beyond a stand-alone variable reference inside "...", an expandable (interpolating) string.

• As such, these two syntax forms are independent of one another and may need to be combined in a given situation; e.g. "$var" / "${var}" work fine, but "$var.someProperty" / "${var}.someProperty" do not: you need "var.someProperty)" / "{var}.someProperty)"

In the context of "...", there is another reason to use ${...}, even if the variable name itself doesn't need it:

If you need to delineate the variable name from directly following non-whitespace characters, notably including ::

$foo = 'bar'  # example variable

# INCORRECT: PowerShell assumes that the variable name is 'foobarian', not 'foo'
PS> "A $foobarian."
A .  # Variable $foobarian doesn't exist -> reference expanded to empty string.

# CORRECT: Use {...} to delineate the variable name:
PS> "A ${foo}barian."
A barbarian.

# INCORRECT: PowerShell assumes that 'foo:' is a *namespace* (drive) reference
#            (such as 'env:' in $env:PATH) and FAILS:
PS> "$foo: bar"
Variable reference is not valid. ':' was not followed by a valid variable name character. 
Consider using ${} to delimit the name.

# CORRECT: Use {...} to delineate the variable name:
PS> "${foo}: bar"
bar: bar

See this answer for a comprehensive overview of PowerShell string-expansion rules.

Note that you need the same technique when string expansion is implicitly applied, in the context of passing an unquoted argument to a command; e.g.:

# INCORRECT: The argument is treated as if it were enclosed in "...",
#            so the same rules apply.
Write-Output $foo:/bar

# CORRECT
Write-Output ${foo}:/bar

Finally, a somewhat obscure alternative is to `-escape the first character after the variable name, but the problem is that this only works as expected with characters that aren't part of escape sequences (see about_Special_Characters):

# OK: because `: is not an escape sequence.
PS> "$foo`: bar"
bar: bar

# NOT OK, because `b is the escape sequence for a backspace character.
PS> "$foo`bar"
baar # The `b "ate" the trailing 'r' of the variable value
     # and only "ar" was the literal part.

tl;dr

Inside "...":

• $(...) is only needed to embed entire expressions or commands (e.g. "PSVersionTable.PSVersion)")

• ${...} is only needed if a stand-alone reference to a variable with a non-special name needs delimiting (disambiguation), so that subsequent characters in the string aren't interpreted as part of the variable name (e.g, "${foo}_bar" correctly embeds variable $foo; without {...}, PowerShell would look for variable $foo_bar; notably, a subsequent : requires this technique too: "${foo}:bar"; by contrast, something like "$foo!" also works without {...})

Independently of use in "...", ${...} is also needed for variable names containing special characters (e.g. ${a-b} = 42)

For a comprehensive overview of PowerShell's expandable strings (string interpolation), see this answer.

Background information:

To complement marsze's helpful answer:

${...} (enclosing the variable name in { and }) is indeed always necessary if a variable name contains special characters, such as spaces, . or -.

• Not special are _ and - surprisingly and problematically - ?.

  • For the exact rules, see the relevant section of the conceptual "about_Variables" help topic, Variable Names that Include Special Characters.

• (The first occurrence of) : in a name is invariably interpreted as terminating either a PowerShell drive reference - in the context of namespace variable notation - or a scope specifier, irrespective of whether {...} enclosure is used or required (e.g., in $env:USERNAME or ${env:USERNAME}, env refers to the Env: PowerShell drive representing all environment variables; in $script:foo or ${script:foo}, script refers to the script's scope and its variables).

Note:

• ${...} - the syntax for disambiguating a variable name - is not to be confused with $(...), the subexpression operator, which is needed to embed any expression or command that goes beyond a stand-alone variable reference inside "...", an expandable (interpolating) string.

• As such, these two syntax forms are independent of one another and may need to be combined in a given situation; e.g. "$var" / "${var}" work fine, but "$var.someProperty" / "${var}.someProperty" do not: you need "var.someProperty)" / "{var}.someProperty)"

In the context of "...", there is another reason to use ${...}, even if the variable name itself doesn't need it:

If you need to delineate the variable name from directly following non-whitespace characters, notably including ::

$foo = 'bar'  # example variable

# INCORRECT: PowerShell assumes that the variable name is 'foobarian', not 'foo'
PS> "A $foobarian."
A .  # Variable $foobarian doesn't exist -> reference expanded to empty string.

# CORRECT: Use {...} to delineate the variable name:
PS> "A ${foo}barian."
A barbarian.

# INCORRECT: PowerShell assumes that 'foo:' is a *namespace* (drive) reference
#            (such as 'env:' in $env:PATH) and FAILS:
PS> "$foo: bar"
Variable reference is not valid. ':' was not followed by a valid variable name character. 
Consider using ${} to delimit the name.

# CORRECT: Use {...} to delineate the variable name:
PS> "${foo}: bar"
bar: bar

See this answer for a comprehensive overview of PowerShell string-expansion rules.

Note that you need the same technique when string expansion is implicitly applied, in the context of passing an unquoted argument to a command; e.g.:

# INCORRECT: The argument is treated as if it were enclosed in "...",
#            so the same rules apply.
Write-Output $foo:/bar

# CORRECT
Write-Output ${foo}:/bar

Finally, a somewhat obscure alternative is to `-escape the first character after the variable name, but the problem is that this only works as expected with characters that aren't part of escape sequences (see about_Special_Characters):

# OK: because `: is not an escape sequence.
PS> "$foo`: bar"
bar: bar

# NOT OK, because `b is the escape sequence for a backspace character.
PS> "$foo`bar"
baar # The `b "ate" the trailing 'r' of the variable value
     # and only "ar" was the literal part.

PowerShell only supports interpolation at the time of a variable creation, unless you are talking about calculated object properties:

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_calculated_properties?view=powershell-7

If you are looking for f-strings see here: https://ss64.com/ps/syntax-f-operator.html