Yes, there is a Spring way of doing this:

Step 1. Adding an additional Tomcat connector

To add a port to the embedded server an additional connector needs to be configured. We will do it by providing custom WebServerFactoryCustomizer:

@Component
public class TomcatContainerCustomizer implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {

    @Value("${swagger.port}")
    private int swaggerPort;

    @Override
    public void customize(TomcatServletWebServerFactory factory) {

        Connector swaggerConnector = new Connector();
        swaggerConnector.setPort(swaggerPort);
        factory.addAdditionalTomcatConnectors(swaggerConnector);
    }
}

Now Tomcat listens on two ports but it serves the same content on both of them. We need to filter it.

Step 2. Adding a filter

Adding a servlet filter is pretty straightforward with a FilterRegistrationBean. It can be created anywhere, I added it directly to the TomcatContainerCustomizer.

@Component
public class TomcatContainerCustomizer  implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {

    @Value("${swagger.port}")
    private int swaggerPort;

    @Value("${swagger.paths}")
    private List<String> swaggerPaths;

    @Override
    public void customize(TomcatServletWebServerFactory factory) {

        Connector swaggerConnector = new Connector();
        swaggerConnector.setPort(swaggerPort);
        factory.addAdditionalTomcatConnectors(swaggerConnector);
    }

    @Bean
    public FilterRegistrationBean<SwaggerFilter> swaggerFilterRegistrationBean() {

        FilterRegistrationBean<SwaggerFilter> filterRegistrationBean = new FilterRegistrationBean<>();
        filterRegistrationBean.setFilter(new SwaggerFilter());
        filterRegistrationBean.setOrder(-100);
        filterRegistrationBean.setName("SwaggerFilter");

        return filterRegistrationBean;
    }

    private class SwaggerFilter extends OncePerRequestFilter {

        private AntPathMatcher pathMatcher = new AntPathMatcher();

        @Override
        protected void doFilterInternal(HttpServletRequest httpServletRequest,
                                        HttpServletResponse httpServletResponse,
                                        FilterChain filterChain) throws ServletException, IOException {

            boolean isSwaggerPath = swaggerPaths.stream()
                    .anyMatch(path -> pathMatcher.match(path, httpServletRequest.getServletPath()));
            boolean isSwaggerPort = httpServletRequest.getLocalPort() == swaggerPort;

            if(isSwaggerPath == isSwaggerPort) {
                filterChain.doFilter(httpServletRequest, httpServletResponse);
            } else {
                httpServletResponse.sendError(404);
            }
        }
    }
}

The properties swagger.port and swagger.paths are configured in the application.yaml:

server.port: 8080
swagger:
  port: 8088
  paths: |
    /swagger-ui.html,
    /webjars/springfox-swagger-ui/**/*,
    /swagger-resources,
    /swagger-resources/**/*,
    /v2/api-docs

So far so good: the swagger-ui is served on the port 8088, our api on the 8080. But there is a problem: when we try to connect to the api from the swagger-ui, the requests are sent to the 8088 instead of 8080.

Step 3. Adjusting SpringFox config.

Swagger assumes that the api runs on the same port as the swagger-ui. We need to explicitly specify the port:

@Value("${server.port}")
private int serverPort;

@Bean
public Docket docket() {
    return new Docket(DocumentationType.SWAGGER_2)
            .host("localhost:" + serverPort);
}

And the last problem: as the ui runs on a different port than the api, the requests are considered cross-origin. We need to unblock them. It can be done globally:

@Bean
public WebMvcConfigurer corsConfigurer() {
    return new WebMvcConfigurer() {
        @Override
        public void addCorsMappings(CorsRegistry registry) {
            registry.addMapping("/**/*").allowedOrigins("http://localhost:" + swaggerPort);
        }
    };
}

or by adding annotations to the controllers:

@CrossOrigin(origins = "http://localhost:${swagger.port}")

Versions used: SpringBoot 2.2.2.RELEASE, springfox-swagger2 2.9.2

For a working example see https://github.com/mafor/swagger-ui-port

@Bean
public Docket categoryApi() {
    return new Docket(DocumentationType.SWAGGER_2)
            .host("twogoods.cc")
            .groupName("bookplatform-api")
            .apiInfo(apiInfo())
            .select()
            .paths(apiPaths())
            .build()
            .directModelSubstitute(java.sql.Timestamp.class, java.sql.Date.class)
            .enableUrlTemplating(false);
}

host() method!!!

I know it's not the question you're asking, but springfox is currently having issues with newer versions of spring. The spring version you're using is still working but as of 2.6 there are bugs, and it looks like the project is not well maintained. Since you're at the beginning of the project, switching is not too hard. You could move to springdocs for example (for migration: https://springdoc.org/#migrating-from-springfox).

With respect to opening a url, there are some good solutions mentioned here: How to open the default webbrowser using java . You could make your swagger url a property and have swagger configure it accordingly, then you can reuse the property to call the url on run-time. If you want to differentiate between environments I'd suggest use profiles. Only open the url in the browser if you start the app on dev environment, and not on prod is then specified by using @Profile("dev"). Create a commandline/application runner with the profile annotation (https://www.tutorialspoint.com/spring_boot/spring_boot_runners.htm), and call the url from there.

That said, combining it gives:

package com.example.demo;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Component;

import java.awt.*;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;

@Profile("dev")
@Component
public class SwaggerRunner implements ApplicationRunner {

    @Value("${springdoc.swagger-ui.path}")
    private String swaggerPath;

    @Override
    public void run(ApplicationArguments args) throws Exception {
        log("\nWelcome to Multi Brow Pop.\nThis aims to popup a browsers in multiple operating systems.\nGood luck!\n");

        final String swaggerUrl = "http://localhost:8000/" + swaggerPath; 
        
        log("We're going to this page: " + swaggerUrl);

        String myOS = System.getProperty("os.name").toLowerCase();
        log("(Your operating system is: " + myOS + ")\n");

        try {
            if (Desktop.isDesktopSupported()) { // Probably Windows
                log(" -- Going with Desktop.browse ...");
                Desktop desktop = Desktop.getDesktop();
                desktop.browse(new URI(swaggerUrl));
            } else { // Definitely Non-windows
                Runtime runtime = Runtime.getRuntime();
                if (myOS.contains("mac")) { // Apples
                    log(" -- Going on Apple with 'open'...");
                    runtime.exec("open " + swaggerUrl);
                } else if (myOS.contains("nix") || myOS.contains("nux")) { // Linux flavours
                    log(" -- Going on Linux with 'xdg-open'...");
                    runtime.exec("xdg-open " + swaggerUrl);
                } else
                    log("I was unable/unwilling to launch a browser in your OS :( #SadFace");
            }
            log("\nThings have finished.\nI hope you're OK.");
        } catch (IOException | URISyntaxException eek) {
            log("**Stuff wrongly: " + eek.getMessage());
        }
    }

    private static void log(String log) {
        System.out.println(log);
    }

}

put springdoc.swagger-ui.path=/custom/path in your application.properties to change the path to your swagger-ui

I had given up and went to use Spring Boot 2.7 after posting the question. But, after seeing Dmitriy's answer though, I checked Springdoc one last time and found that Springdoc v2 does support Spring Boot 3.

Essentially, one has to place the following in their pom:

<dependency>
   <groupId>org.springdoc</groupId>
   <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
   <version>2.0.0</version>
</dependency>

Then one can access the Swagger page using the following URL: http://localhost:8080/swagger-ui.html (Don't forget to add context path if you need it). For some reason, when opening, it redirects to http://localhost:8080/swagger-ui/index.html although going for that initially returned 404.