If you come from Stack Overflow, using atan2 might be a simpler solution for you.

Let , . If is the "oriented" angle from to (that is, rotating by gives ), then:

In Matlab, this is equivalent to wrapToPi(angle(x2+i*y2) - angle(x1+i*y1)).

Your statement that

the angle between two vectors is supposed to be their inner product

is incorrect, as is the statement from the book. On the Wikipedia page on the dot product, you can see the correct formula for the angle between two complex vectors $u$ and $v$ (thanks to Henry for catching the earlier mistake): $$\theta=\arccos\left(\frac{\operatorname{Re}(u\cdot v)}{\|u\|\|v\|}\right)$$ where the inner product $u\cdot v$ is defined to be $$u\cdot v=\sum_{k=0}^{n-1} u_k\overline{v_k}$$ I would guess that perhaps the intended meaning of the "scaling factor" is as follows: when $u$ and $v$ are unit vectors, we have $$\cos(\theta)=\operatorname{Re}(u\cdot v)$$ while when $u$ and $v$ are arbitrary non-zero vectors, we have $$\cos(\theta)=\frac{\operatorname{Re}(u\cdot v)}{\|u\|\|v\|}$$ (the quantities $\|u\|$ and $\|v\|$ are both equal to $1$ when $u$ and $v$ are unit vectors). This would make $$\frac{1}{\|u\|\|v\|}$$ the "scaling factor", though it is scaling the formula for the cosine of the angle, not the angle itself.

2D case

Just like the dot product is proportional to the cosine of the angle, the determinant is proportional to its sine. So you can compute the angle like this:

dot = x1*x2 + y1*y2      # Dot product between [x1, y1] and [x2, y2]
det = x1*y2 - y1*x2      # Determinant
angle = atan2(det, dot)  # atan2(y, x) or atan2(sin, cos)

The orientation of this angle matches that of the coordinate system. In a left-handed coordinate system, i.e. x pointing right and y down as is common for computer graphics, this will mean you get a positive sign for clockwise angles. If the orientation of the coordinate system is mathematical with y up, you get counterclockwise angles as is the convention in mathematics. Changing the order of the inputs will change the sign, so if you are unhappy with the signs just swap the inputs.

3D case

In 3D, two arbitrarily placed vectors define their own axis of rotation, perpendicular to both. That axis of rotation does not come with a fixed orientation, which means that you cannot uniquely fix the direction of the angle of rotation either. One common convention is to let angles be always positive, and to orient the axis in such a way that it fits a positive angle. In this case, the dot product of the normalized vectors is enough to compute angles.

dot = x1*x2 + y1*y2 + z1*z2    # Between [x1, y1, z1] and [x2, y2, z2]
lenSq1 = x1*x1 + y1*y1 + z1*z1
lenSq2 = x2*x2 + y2*y2 + z2*z2
angle = acos(dot/sqrt(lenSq1 * lenSq2))

Note that some comments and alternate answers advise against the use of acos for numeric reasons, in particular if the angles to be measured are small.

Plane embedded in 3D

One special case is the case where your vectors are not placed arbitrarily, but lie within a plane with a known normal vector n. Then the axis of rotation will be in direction n as well, and the orientation of n will fix an orientation for that axis. In this case, you can adapt the 2D computation above, including n into the determinant to make its size 3×3.

dot = x1*x2 + y1*y2 + z1*z2
det = x1*y2*zn + x2*yn*z1 + xn*y1*z2 - z1*y2*xn - z2*yn*x1 - zn*y1*x2
angle = atan2(det, dot)

One condition for this to work is that the normal vector n has unit length. If not, you'll have to normalize it.

As triple product

This determinant could also be expressed as the triple product, as @Excrubulent pointed out in a suggested edit.

det = n · (v1 × v2)

This might be easier to implement in some APIs, and gives a different perspective on what's going on here: The cross product is proportional to the sine of the angle, and will lie perpendicular to the plane, hence be a multiple of n. The dot product will therefore basically measure the length of that vector, but with the correct sign attached to it.

Range 0 – 360°

Most atan2 implementations will return an angle from [-π, π] in radians, which is [-180°, 180°] in degrees. If you need positive angles [0, 2π] or [0°, 360°] instead, you can just add 2π to any negative result you get. Or you can avoid the case distinction and use atan2(-det, -dot) + π unconditionally. If you are in a rare setup where you need the opposite correction, i.e. atan2 returns non-negative [0, 2π] and you need signed angles from [-π, π] instead, use atan2(-det, -dot) - π. This trick is actually not specific to this question here, but can be applied in most cases where atan2 gets used. Remember to check whether your atan2 deals in degrees or radians, and convert between these as needed.