In this simple case, you could get by with using

sum(x.*x)

It seems times (.*) is supported properly for integer matrices, although mtimes ( *) is not.

For general matrix multiplication: let A and B be two matrices with suitable sizes so that A*B exists. Since times and sum are supported for integers, you can generalize the above trick, usingbsxfun and sum to compute all entries of the product matrix as follows.

Edit: As noted by @July, you need the 'native' flag in sum in order to keep the result of integer type. Thanks also for pointing out a problem that was caused by squeeze, now corrected by using a second permute.

permute(sum(bsxfun(@times, A.', permute(B, [1 3 2])), 1, 'native'), [2 3 1])

For example:

>> A = int64([1 2; 3 4])
A =
                    1                    2
                    3                    4
>> B = int64([5 7 9; 6 8 10])
B =
                    5                    7                    9
                    6                    8                   10
>> permute(sum(bsxfun(@times, A.', permute(B, [1 3 2])), 'native'), [2 3 1])
ans =
    17    23    29
    39    53    67

Anyway, the fastest alternative seems to be double(A)*double(B).