Definitely the second method is preferred because you don't have the overhead of another function invocation:

window.location.href = "webpage.htm";

Button has a type attribute which defaults to submit: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/button#attr-type
While this does not affect "everyday" buttons, if the button resides in a form, this way it will submit the form, and result in some page loading, which clashes with your own attempt. You can just add a type="button" attribute to the button to avoid that:

<button id="rgstr_btn" type="button" class="btn btn-info" onClick="store()">Register</button>
                       ^^^^^^^^^^^^^

On Windows the popup isn't closed automatically after a new window is opened from a link within the popup.

Close it in the click handler manually, this won't hurt Linux but will help on Windows:

document.addEventListener("click", function(event) {
    if (event.target.localName == "a") {
        close();
    }
});

The related questions linked in your question don't apply here as the first is for userscripts, not extensions, and the second isn't for popups shown by the browser when you click the toolbar button.

gBrowser.loadURI('http://www.example.com');

works properly.

I suspect you can remove tryOpenTab2 from the submitting process and just have the form submit as normal without a target

Else try this

Change

if (w && !w.closed) {
  //w.close();// client want to always open new tab
    w = null;
}
wId = ''+new Date().getTime();
w = window.open("",'myPopup_Window' + wId); 
form.attr('target', 'myPopup_Window' + wId);

to

 form.attr('target', '_self');