TL;DR: You need to use the inverse of the transformation matrix in your question.
You’ve made a fairly common error here. The transformation matrix that you’ve shown maps the old coordinate axes onto the new ones. However, to get the coordinates of a point relative to these new axes, you have to invert that transformation. Why this is so is covered elsewhere on this SE and on the Internet, but I’ll briefly explain why here.
Let 
and 
be the positive unit direction vectors of an arbitrary pair of coordinate axes, and 
the origin point of this coordinate system. We can express any point as 
; the coefficients 
and 
are the coordinates of this point in this coordinate system. The 
- and 
- coordinates in the standard coordinate system can be understood in the same way by taking 
and 
, so that we have 
. We can write 
and 
, and if we express 
as coordinates in the standard coordinate system, we get the following identity: 
Using homogeneous coordinates, we can express this identity in matrix form as $$\begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}a&c&O_x\\b&d&O_y\\0&0&1\end{bmatrix} \begin{bmatrix}u\\v\\1\end{bmatrix}.$$ Note that the 
matrix in this identity maps the standard unit coordinate vectors onto the new ones and sends the origin to 
. We want to solve this equation for 
and 
, which we can do by multiplying both sides by the inverse of the 
matrix.
In your case, 
gets sent to 
, 
gets sent to 
, and the origin gets sent to 
. The correct matrix for computing the new coordinates of a point is therefore 
More generally, if the new coordinate axes are obtained by rotating and translating the old ones, the coordinate transformation matrix will be 
Here I used the fact that the inverse of a rotation matrix is its transpose.
Videos
How do I use the matrix transformation calculator?
How are the transformations applied?
What transformations can I apply?
TL;DR: You need to use the inverse of the transformation matrix in your question.
You’ve made a fairly common error here. The transformation matrix that you’ve shown maps the old coordinate axes onto the new ones. However, to get the coordinates of a point relative to these new axes, you have to invert that transformation. Why this is so is covered elsewhere on this SE and on the Internet, but I’ll briefly explain why here.
Let and be the positive unit direction vectors of an arbitrary pair of coordinate axes, and the origin point of this coordinate system. We can express any point as ; the coefficients and are the coordinates of this point in this coordinate system. The - and - coordinates in the standard coordinate system can be understood in the same way by taking and , so that we have . We can write and , and if we express as coordinates in the standard coordinate system, we get the following identity: Using homogeneous coordinates, we can express this identity in matrix form as $$\begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}a&c&O_x\\b&d&O_y\\0&0&1\end{bmatrix} \begin{bmatrix}u\\v\\1\end{bmatrix}.$$ Note that the matrix in this identity maps the standard unit coordinate vectors onto the new ones and sends the origin to . We want to solve this equation for and , which we can do by multiplying both sides by the inverse of the matrix.
In your case, gets sent to , gets sent to , and the origin gets sent to . The correct matrix for computing the new coordinates of a point is therefore More generally, if the new coordinate axes are obtained by rotating and translating the old ones, the coordinate transformation matrix will be Here I used the fact that the inverse of a rotation matrix is its transpose.
I can't really comment due to reputation but, can you please include the details: what exactly you're trying to do and include a grid of reference to denote the transformation you're trying to achieve(or atleast the position of origin) and label the vertices of the image in the rectangle. I'll try to help you further then. e.g. an explanation like rotating the rectangle and translating via xyz.