The problem described can be solved as follows. Let

M = m_11 m_12 m_13
    m_21 m_22 m_23
    m_31 m_32 m_33

denote the desired rotation matrix. We require

 1 0 0 * M + t = x_x x_y x_z
 0 1 0           y_x y_y y_z
 0 0 1           z_x z_y z_y

where t denotes the translation; we see that this matrix equality can be solved by multiplying from the left with the identity matrix, which is the inverse of itself; hence we obtain the following equality.

 M + t = x_x x_y x_z
         y_x y_y y_z
         z_x z_y z_y

This can be rearranged by subtracting t from both sides to obtain the desired matrix M as follows.

 M = x_x x_y x_z - t = x_x-t_x x_y-t_y x_z-t_z 
     y_x y_y y_z       y_x-t_x y_y-t_y y_z-t_z
     z_x z_y z_y       z_x-t_x z_y-t_y z_z-t_z

Note that this was relatively easy as the initial matrix consists out of the basic vectors of the standard base. In general it is more difficult and involves a basis transformation, which basically can be done by Gaussian elimination, but can be numerically difficult.

(1) is correct.

It is good to have a deeper understanding of the objects involved here. Let’s say we are talking about a three-dimensional vector space . Then a coordinate system consists of three linearly independent vectors, e.g. $A = (a_1, a_2, a_3), a_i ∈ V$. It gives a natural isomorphism $$ℝ^3 ∋ \begin{pmatrix}x\\y\\z\end{pmatrix} ↦ xa_1+ya_2+za_2 ∈ V,$$ which I will denote by .

In the case where , the coordinate system can be thought of as a matrix, where the basis vectors are the column vectors. Then the isomorphism $\tilde{A}: ℝ^3 → V = ℝ^3$ is given by left-multiplication with that matrix.

Notice however that the operation “write a vector in coordinates with respect to coordinate system A” is given by the inverse of the isomorphism . So for this is left-multiplication by , the inverse matrix of .

Now your question becomes: which matrix , acting on the coordinate representation under gives the coordinate representation of ? According to the above note that means $$R ∘ \tilde{A}^{-1} = \tilde{B}^{-1}$$ and in the case this is equivalent to $$R⋅A^{-1} = B^{-1} ⇔ A = B ⋅ R.$$

Advanced Note: When thinking about a manifold , the same reasoning leads to the right action of on the frame bundle .

One easy way is to think of both coordinate systems as transforms from the unit vectors (1,0,0) (0,1,0) and (0,0,1). You start off in this coordinate space (I will call it '1')whose transform matrix is the identity matrix:

    [1,0,0]
I = [0,1,0]
    [0,0,1]

then your first coordinate space (I will call it '2') has the transform matrix:

    [Xx,Xy,Xz]
A = [Yx,Yy,Yz]
    [Zx,Zy,Zz]

and your second coordinate space (I will call it '3') has the transform matrix:

    [Xx',Xy',Xz']
B = [Yx',Yy',Yz']
    [Zx',Zy',Zz']

For your points to be in the first coordinate system, then you have transformed them from 1 to 2. If you want to go from 2 to 3 then you can undo the transform from 1 to 2 then do the transform from 1 to 3. You can reverse the transform by inverting 2's transform matrix.

A point v in 2 can be transformed to a point v' in 3 with this equation: v' = B(A^-1)v where (A^-1) is the inverse of A.

Note this also handles scaling even though you don't need it. This approach will work with translation as well, though you would need a 4x4 matrix instead of a 3x3.