You can use conda remove --force.

The documentation says:

--force               Forces removal of a package without removing packages
                      that depend on it. Using this option will usually
                      leave your environment in a broken and inconsistent
                      state

Conda's remove operation still needs to satisfy all the other specifications for the environment, so Conda invokes its solver and this can be complicated. Essentially, it re-solves the entire environment sans the specified package, compares that against the existing state, then makes a plan based on the difference.

I very much doubt there is anything directly impactful about size of package, which OP alludes to. Instead, things that negatively impact solving are:

• having a large environment (e.g., anaconda package is installed)
• channel mixing - in particular, including the conda-forge channel at equal or higher priority as defaults in an environment with the anaconda package; that package and all its dependencies are intended to be sourced from the anaconda channel
• having an underspecified environment (see conda env export --from-history to see your explicit specifications); e.g., an environment with a python=3.8 specification will be easier on the solver than just a python specification

In general, using smaller specialized (e.g., per-project) environments, rather than large monolithic ones helps avoid such problems. The anaconda package is particularly problematic.

Try Mamba

Other than adopting better practices, one can also get significantly faster solves with Mamba, a drop-in compiled replacement for conda. Try it out:

## install Mamba in base env
conda install -n base conda-forge::mamba

## use it like you would the 'conda' command
mamba remove -n foo bar