vscode can only use one service as "the workspace" where the IDE runs. Just like when working locally you are on the IDE and the other services run in other containers.

None of your current services seem "good" for being the IDE workspace, so you would have to add that one. That would be "just like" your host machine, but in the container.

You can use multiple compose files so you can avoid changing your current docker-compose.yml while being able to add your new service.

So, part I:

1. Create a second docker-compose.yml file (maybe: docker-compose.workspace.yml)
2. Add one single service to that file, maybe call it "workspace". vscode Remote part will run there. What image? You may use one vscode's precooked ones.
3. On .devcontainer.json point to both files and define the workspace service:

...
  "dockerComposeFile": [
    "docker-compose.yaml",
    "docker-compose.workspace.yaml"
  ],
  "service": "workspace",
...

Ok. So that gives you the workspace container and everything else on the side. That gets us to the other part of the question:

Is it possible to interact with Docker engine on the host from inside the container? So I can simply have a dev.sh script that can simply up the rabbitmq and influxdb dependencies and then launch whatever microservice I want to run?

First, if you want the docker & docker-compose commands you have to install the packages. Some images have them builtin, others not. You may use your own image, etc.

But that is not enough. You workspace container is ignorant of the host's docker. But it is easy enough to fix. Just add a volume mount:

/var/run/docker.sock:/var/run/docker.sock

On the workspace service. That way vscode will "see" your host's docker and operate with it.

Beware that it is still the host's docker, so you may get into trouble with paths, etc depending on what you do.

After more research, I have found a decent solution. This solution makes use of multi-stage builds in Docker. This stackoverflow answer gave me the clue that got me pointed in the right direction.

The first change was to add multi-stage builds to the existing Dockerfile.

FROM python:3.11 as base
# Do the original image build instructions

FROM base as dev
# Do the build steps needed to add any 
#  development tools or configuration

Then, in the original docker-compose.yml file, set the build target to be the base image.

services:
  web:
    build:
     ...
     target: base
    ...

Then, in the .devcontainer/docker-compose.yml file, set the build target to be the dev image.

services:
  web:
    build:
     ...
     target: dev
    ...

Now, when building the images, you can specify which target to build just by changing the docker-compose files included at the command line. To build for deployment, use docker compose -f docker-compose.yml.

{
    "name": "Existing Docker Compose (Extend)",
    // Update the 'dockerComposeFile' list if you have more compose files or use different names.
    // The .devcontainer/docker-compose.yml file contains any overrides you need/want to make.
    "dockerComposeFile": [
        "../docker-compose.yml",
        "docker-compose.yml"
    ],
    ...

The .devcontainer/devcontainer.json file will specify to include both docker compose files. vscode will now build the dev container which includes the additional development tools and configuration.