In certain cases, by checking the return value for NULL also gives an indication of whether the function succeeded or not. Not necessarily ideal practise however.

In C the function strcat does not create a new character array containing concatenated strings. It appends characters from the second string to the first string of the first character array provided that it has enough elements to store the new characters. Otherwise the function will try to overwrite the memory beyond the character array that results in undefined behavior.

So a valid use of the function in the first program can look the following way

#include <stdio.h>
#include <string.h>

int main(int argc, const char *argv[]) {
    char s1[11] = "12345";
    char s2[] = "abcde";

    strcat(s1, s2);

    puts(s1);
    puts(s2);
    return 0;
} 

In this program the character array is declared as having 11 elements. Thus it is able to accommodate the appended string "abcde".

In the second program there is an attempt to modify the string literal pointed to by the pointer s1. String literals in C and C++ are immutable. Any attempt to change a string literal results in undefined behavior even though in C opposite to C++ string literals have types of non-constant character arrays.

From the C Standard (6.4.5 String literals)

7 It is unspecified whether these arrays are distinct provided their elements have the appropriate values. If the program attempts to modify such an array, the behavior is undefined.

So in the second program you again need to use a character array with enough elements. For example

#include <stdio.h>
#include <string.h>

int main(int argc, const char *argv[]) {
    char s1[11] = "12345";
    char* s2 = "abcde";

    strcat(s1, s2);

    puts(s1);
    puts(s2);
    return 0;
}

Or you could use either a Variable Length Array (VLA) if the compiler supports them or dynamically allocate an array. For example

#include <stdio.h>
#include <string.h>

int main(int argc, const char *argv[]) {
    char *s1 = "12345";
    char* s2 = "abcde";
    char s3[strlen( s1 ) + strlen( s2 ) + 1];    

    strcpy( s3, s1 );
    strcat( s3, s2 );

    puts(s1);
    puts(s2);
    puts(s3);

    return 0;
}

Or

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main(int argc, const char *argv[]) {
    char *s1 = "12345";
    char* s2 = "abcde";
    char *s3 = malloc( strlen( s1 ) + strlen( s2 ) + 1 );    

    if ( s3 != NULL )
    {
        strcpy( s3, s1 );
        strcat( s3, s2 );

        puts(s1);
        puts(s2);
        puts(s3);
    }

    free( s3 );

    return 0;
}

const char *src

src shouldn't be modified by the function, hence use const correctness to mark it as read-only.

char *rdest = dest;

Save the original position until later, since there's a requirement that strcat should return a pointer to the first element of the merged string (return rdest;).

while (*dest)
dest++;

The while loop is implicitly looking for the null terminator. Meaning: find the end of the first string, so that after this loop, dest points at the null terminator of that string.

while (*dest++ = *src++)

This is a common, although admittedly confusing idiom in C. (It actually implements strcpy in this line.) Operator precedence says postfix ++ takes precedence over prefix * over assignment =.

So first each pointer is evaluated and ++ is applied to the pointers, not the pointed-at data. But since it is postfix, the actual increment of the pointer address does not happen until the end of the expression.

* takes the contents of each pointer before this increment, and then = copies the content from *src to *dest. Again, this happens before the addresses are incremented.

Finally, there is an implicit check against null termination, since the result of the = operand can actually be checked - it is equivalent to its left operand, in this case *dest. And note that the null terminator gets copied, too.

You could rewrite this while loop in a less confusing way:

*dst = *src;
while(*src != '\0')
{
  dst++;
  src++;
  *dst = *src;
}