You might look at jqueryui. That projects provides you with modal dialogs that should do what you want.

http://jqueryui.com/demos/dialog/

From the jqueryui site:

A call to $(foo).dialog() will initialize a dialog instance and will auto-open the dialog by default. If you want to reuse a dialog, the easiest way is to disable the "auto-open" option with: $(foo).dialog({ autoOpen: false }) and open it with $(foo).dialog('open'). To close it, use $(foo).dialog('close')

First, don't wrap your form inside the success div, keep it separate like this:

<div id="successMessage"></div>

Next, in the ajax success function just do this to hide the form and display message:

success: function(msg){
         if(msg=="Updated Successfully")
         {
             $("#successMessage").html(msg);
             $("#update_form").hide();
         }
         else
         {
             $("#errorMessage").html(msg);
             $("#errorMessageDialog").dialog();
         }
}

For the error message it is a little bit trickier. First change the error message div to something like this:

<div style="display: none;" id="errorMessageDialog">
<p id="errorMessage"></p>
</div>

Then in the ajax error function open set the error message and open the dialog like this:

error:function(msg){
      $("#errorMessage").html(msg);
      $("#errorMessageDialog").dialog();
}

You can use the errorPlacement option to override the error message display with little css. Because css on its own will not be enough to produce the effect you need.

$(document).ready(function(){
    $("#myForm").validate({
        rules: {
            "elem.1": {
                required: true,
                digits: true
            },
            "elem.2": {
                required: true
            }
        },
        errorElement: "div",
        wrapper: "div",  // a wrapper around the error message
        errorPlacement: function(error, element) {
            offset = element.offset();
            error.insertBefore(element)
            error.addClass('message');  // add a class to the wrapper
            error.css('position', 'absolute');
            error.css('left', offset.left + element.outerWidth());
            error.css('top', offset.top);
        }

    });
});

You can play with the left and top css attributes to show the error message on top, left, right or bottom of the element. For example to show the error on the top:

    errorPlacement: function(error, element) {
        element.before(error);
        offset = element.offset();
        error.css('left', offset.left);
        error.css('top', offset.top - element.outerHeight());
    }

And so on. You can refer to jQuery documentation about css for more options.

Here is the css I used. The result looks exactly like the one you want. With as little CSS as possible:

div.message{
    background: transparent url(msg_arrow.gif) no-repeat scroll left center;
    padding-left: 7px;
}

div.error{
    background-color:#F3E6E6;
    border-color: #924949;
    border-style: solid solid solid none;
    border-width: 2px;
    padding: 5px;
}

And here is the background image you need:

(source: scriptiny.com)

If you want the error message to be displayed after a group of options or fields. Then group all those elements inside one container a 'div' or a 'fieldset'. Add a special class to all of them 'group' for example. And add the following to the begining of the errorPlacement function:

errorPlacement: function(error, element) {
    if (element.hasClass('group')){
        element = element.parent();
    }
    ...// continue as previously explained

If you only want to handle specific cases you can use attr instead:

if (element.attr('type') == 'radio'){
    element = element.parent();
}

That should be enough for the error message to be displayed next to the parent element.

You may need to change the width of the parent element to be less than 100%.

I've tried your code and it is working perfectly fine for me. Here is a preview:

I just made a very small adjustment to the message padding to make it fit in the line:

div.error {
    padding: 2px 5px;
}

You can change those numbers to increase/decrease the padding on top/bottom or left/right. You can also add a height and width to the error message. If you are still having issues, try to replace the span with a div

<div class="group">
<input type="radio" class="checkbox" value="P" id="radio_P" name="radio_group_name"/>
<label for="radio_P">P</label>
<input type="radio" class="checkbox" value="S" id="radio_S" name="radio_group_name"/>
<label for="radio_S">S</label>
</div>

And then give the container a width (this is very important)

div.group {
    width: 50px; /* or any other value */
}

About the blank page. As I said I tried your code and it is working for me. It might be something else in your code that is causing the issue.

Use errorPlacement option

errorPlacement (default: Places the error label after the invalid element) Type: Function() Customize placement of created error labels. First argument: The created error label as a jQuery object. Second argument: The invalid element as a jQuery object.

    errorPlacement: function(error,element){ 
                        //error.insertAfter(element); 
                        //use error.html() as content in your popup div or simply
                        alert(error.html());             
                     }

Yes, that is the correct way.

See the reference here:

http://www.javascriptkit.com/javatutors/error2.shtml

And explanation of how to see more details of the error here:

http://www.javascriptkit.com/javatutors/error3.shtml

Their example:

window.onerror = function(msg, url, linenumber) {
    alert('Error message: '+msg+'\nURL: '+url+'\nLine Number: '+linenumber);
    return true;
}

If you wish to display a LIST of errors in a single pop-up, it's trickier.

Since the errors occue 1 by 1, you need to do the following:

• have window.onerror handler store error details in some array

• Check that array periodically - either via a timer, or on every N'th call of window.onerror handler, or both.

  When the check happens, process entire array, display contents as desired, and empty out an array

You can use apex:pageMessage to display individual message on VF page

This is a kind of static message which you will write in visualforce itself, and can put some conditions in rendered attribute to display the message.

<apex:pageMessage id="mymessage" severity="Error" summary="First Error message on page" strength="1"></apex:pageMessage>

This would render to

severity can be anything from 'confirm', 'info', 'warning', 'error'.

This will render below html code

<div id="page:mymessage:j_id0:j_id2" class="messageText">
    <span id="page:mymessage:j_id0:j_id3">
    <h4></h4>
    </span>
    First Error message on page<br>
</div>

Html Id can very according to your code

You can easily manipulate this code using jQuery and HTML Id.

Example

$("[id*=page:mymessage]").each(function() {
 this.style.display = "none";
 });

To Know more in Detail

You should really look at using the following option for jQuery Validation

showErrors

here is a link to the documentation http://docs.jquery.com/Plugins/Validation/validate#options

You can add the ui-state-error class that comes in your theme, like this:

$("#div-dialog-warning").dialog({
    title: t,
    resizable: false,
    height: 160,
    modal: true,
    buttons: {
        "Ok" : function () {
            $(this).dialog("close");
        }
    }
}).parent().addClass("ui-state-error");

Since the dialog gets wrapped we're using .parent() to get the container including the titlebar. Your theme looks like flick so here's a demo showing that theme in action.