Definitely the second method is preferred because you don't have the overhead of another function invocation:

window.location.href = "webpage.htm";

Button has a type attribute which defaults to submit: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/button#attr-type
While this does not affect "everyday" buttons, if the button resides in a form, this way it will submit the form, and result in some page loading, which clashes with your own attempt. You can just add a type="button" attribute to the button to avoid that:

<button id="rgstr_btn" type="button" class="btn btn-info" onClick="store()">Register</button>
                       ^^^^^^^^^^^^^

On Windows the popup isn't closed automatically after a new window is opened from a link within the popup.

Close it in the click handler manually, this won't hurt Linux but will help on Windows:

document.addEventListener("click", function(event) {
    if (event.target.localName == "a") {
        close();
    }
});

The related questions linked in your question don't apply here as the first is for userscripts, not extensions, and the second isn't for popups shown by the browser when you click the toolbar button.

Definitely the second method is preferred because you don't have the overhead of another function invocation:

window.location.href = "webpage.htm";

gBrowser.loadURI('http://www.example.com');

works properly.

Use _self instead of _blank.

window.open(url, "_self");

1. _blank - URL is loaded into a new window. This is default
2. _parent - URL is loaded into the parent frame
3. _self - URL replaces the current page
4. _top - URL replaces any framesets that may be loaded name - The name of the window

For further details. See This Link

To prevent the page from submitting, you need to do it like this:

function checkcat() { 
    if ($(".caty").find(".active-cat").length > 0){ 
      window.open("http://www.google.com","_self");
    } 
    else {
      alert('Aloha!!!');
    }

    return false;
}

Only return false will do the job. In your method, you do not pass in the event object and calling stopPropagation and preventDefault will throw exception as there are no such methods.

If you call checkcat inside form submit, remember to include the return to stop the form from submitting:

<form onsubmit="return checkcat()">
</form>

I suspect you can remove tryOpenTab2 from the submitting process and just have the form submit as normal without a target

Else try this

Change

if (w && !w.closed) {
  //w.close();// client want to always open new tab
    w = null;
}
wId = ''+new Date().getTime();
w = window.open("",'myPopup_Window' + wId); 
form.attr('target', 'myPopup_Window' + wId);

to

 form.attr('target', '_self');