Undefined typically refers to something which has not yet been assigned a value (yet). Null refers to something which definitively has no value. In that case, I would recommend returning a null. Note that a function with no specified return value implicitly returns undefined.

From the ECMAScript2015 spec

4.3.10 undefined value

primitive value used when a variable has not been assigned a value

4.3.12 null value

primitive value that represents the intentional absence of any object value

http://www.ecma-international.org/ecma-262/6.0/#sec-terms-and-definitions-undefined-type

Further reading:

When is null or undefined used in JavaScript?

Other people have given good, correct answers but I want to be explicit about why, since it might not be obvious to some people (not directed at the OP).

A function is nothing more than a set of steps for the computer to take.

This is known as a function call:

getSmallestDivisor(121)

Anytime the return keyword is used, the function stops and replaces the function call with whatever comes after that return word (it could be nothing).

So in this case, the problem with the original function is that when the script reaches this line...

getSmallestDivisor(xSqrt);

...it returns 11 to that function call, which never gets returned to the original function call that happened inside of alert().

So the solution is simply to add a return before the one where it calls itself.

return getSmallestDivisor(xSqrt);

This is a common mistake when making recursive functions. A good way to help figure out what is going on is to make extensive use of the browser console.

function getSmallestDivisor(xVal) {    
    console.log("This is xVal: " + xVal);
    if (xVal % 2 === 0) {
        console.log("xVal % 2 === 0 was true");
        return 2;
    }
    else if (xVal % 3 === 0) {
        console.log("xVal % 3 === 0 was true");
        return 3;
    }
    else {
        console.log("This is else.");
        var xSqrt = Math.sqrt(xVal);
        console.log("This is xSqrt of xVal: " + xSqrt);
        if (xSqrt % 1 === 0) {
            console.log("xSqrt % 1 === 0 was true... recursing with xSqrt!!!");
            getSmallestDivisor(xSqrt);
        }
        else {
            console.log("This is the else inside of else. I am returning: " + xVal);
            return xVal;
        }
    }
}
var y = getSmallestDivisor(121);
console.log("This is y: " + y);

Now in your browser, you can open the console (Option + Command + I in most browsers on macOS) and watch what is happening - which parts get executed, etc.

Your doSomething() function doesn't return anything, which means an assignment using it will be undefined. But, that's not really the problem here.

The underlying problem is that you seem to be mixing two different data processing patterns here: if you're writing purely synchronous code, then use returning functions (which immediately return some value). If you need asynchronous code, then use a callback (which will "eventually" do something). Mixing those two patterns is a recipe for problems and frustration:

Either:

1. don't name your function a "callback", and have it return its processed value, or
2. make the callback responsible for doing whatever it is you were going to do with val.

Case 1:

function doSomething(data, processor) {
  return processor(data);
}

function passThrough(v) { return v; }

var val = doSomething("test", passThrough);
// immediately use "val" here in for whatever thing you need to do.

Case 2:

function doSomething(data, callback) {
  // _eventually_ a callback happens - for instance, this
  // function pulls some data from a database, which is one
  // of those inherently asynchronous tasks. Let's fake that
  // with a timeout for demonstration purposes:
  setTimemout(() => callback(data), 500);
}

function handleData(val) {
  // use "val" here in for whatever thing you need to do. Eventually.
}

doSomething("test", handleData);

And if you want to go with case 2, you really want to have a look at "Promises" and async/await in modern Javascript, which are highly improved approaches based on the idea of "calling back once there is something to call back about".

2021 edit: a third option since original writing this answer is to use the async/await pattern, which is syntactic sugar around Promises.

Case 3:

async function doSomething(input) {
  // we're still _eventually_ returning something,
  // but we're now exploiting `async` to wrap a promise,
  // which lets us write normal-looking code, even if what
  // we're really doing is returning a Promise object,
  // with the "await" keyword auto-unpacking that for us.
  return someModernAsyncAPI.getThing(input);
}

function handleData(val) {
  // ...
}

async function run() {
  const data = await doSomething("test");
  handleData(data);
}

run();