Maybe this is a possible update:

In previous answers and comments only a SOAP API was mentioned, which is however, as described here, outdated.

A possibility to use the new(?) API is described here:

https://validator.w3.org/nu/?doc=https%3A%2F%2Fwww.w3.org%2F&out=json

Maybe someone can confirm this.

I think the code snippet below will you can get the same effect and user experience you’re after.

It’s written using jQuery’s $.ajax(…) with some DOMParser and document.write(…) to put the styled results and UI of the W3C HTML Checker into a new window the way it seems you want.

var validator_baseurl= "https://validator.w3.org/nu/";
var validator_requesturl = validator_baseurl
    + "?showsource=yes&showoutline=yes";
$.ajax({
    url: validator_requesturl,
    type: "POST",
    crossDomain: true,
    data: storyhtml,
    contentType: "text/html;charset=utf-8",
    dataType: "html",
    success: function (response) {
        var results = (new DOMParser()).parseFromString(response, "text/html");
        results.querySelector("link[rel=stylesheet]").href
            = validator_baseurl + "style.css";
        results.querySelector("script").src
            = validator_baseurl + "script.js";
        results.querySelector("form").action
            = validator_requesturl;
        var newWin = window.open("about:blank",
            "Checker results", "height=825,width=700");
        newWin.document.open();
        newWin.document.write(results.documentElement.outerHTML);
        newWin.document.close();
        newWin.location.hash = "#textarea";
        setTimeout(function() {
            newWin.document.querySelector("textarea").rows = "5";
        }, 1000)
    }
});

Explanation

• causes a POST request to be sent to the W3C HTML Checker
• makes the storyhtml text the POST body
• makes text/html;charset=utf-8 the POST body’s media type (what the checker expects)
• causes the checker to actually check the storyhtml contents automatically
• shows the checker results in a new window right when it’s first opened, in one step (so your users don’t need to do a second step to manually submit it for checking themselves)
• replaces relative URLs for the checker’s frontend CSS+JS with absolute URLs (otherwise in this “standalone window” context, the CSS wouldn’t get applied, and the script wouldn’t run)
• newWin.location.hash = "#textarea" is needed to make the checker show the textarea

Notes

• intentionally uses the current W3C HTML Checker (not the legacy W3C markup validator)

• intentionally sends the content to be checked as a POST body, not multipart/form-data); the checker supports multipart/form-data but making it a POST body is easier and better

• the setTimeout textarea bit isn’t required; I just put it to make the results visible without scrolling (bottom part of new window below textarea); you can of course remove it if you want

• sets the new window’s height and width a bit larger than the 600x600 in the question’s original code; again, I just did that to make things easier to see; change them however you want

• uses standard DOM ops that may have better jQuery methods/idioms (I don’t normally use jQuery, so I can imagine there are ways to streamline the code in it further around JQuery)

• could of course also be done without using jQuery at all—using standard Fetch or XHR instead (and I’d be happy to also add examples here that use Fetch and XHR if desired)

• tested & works as expected in Edge, Firefox, Chrome & Safari; but as with any code that uses document.open, Safari users need to unset Preferences > Security > Block pop-up windows