So it turns out it's pretty easy: replace the "port" and "host" attributes with "connect", so if your configuration was:

{
      "type": "python",
      "request": "attach",
      "name": "attach remote",
      "host": "192.168.1.101",
      "port": 8765,
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}/...",
          "remoteRoot": "/usr/app/..."
        }
      ],
      "justMyCode": false
    },

It becomes:

{
      "type": "debugpy",
      "request": "attach",
      "name": "attach remote",
      "connect": { "host": "192.168.1.101", "port": 8765 },
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}/...",
          "remoteRoot": "/usr/app/..."
        }
      ],
      "justMyCode": false
    },

Did you add a breakpoint?

debugpy.breakpoint()

import debugpy

# 5678 is the default attach port in the VS Code debug configurations. Unless a host and port are specified, host defaults to 127.0.0.1
debugpy.listen(5678)
print("Waiting for debugger attach")
debugpy.wait_for_client()
debugpy.breakpoint() #must have
print('break on this line')

Ref: https://code.visualstudio.com/docs/python/debugging

Turns out I needed to download a Python debugger in the VSCode extensions. I used Python Debugger from Microsoft.