See this newsgroup thread where Bruno Luong gives advice on how to do this calculation without using the symbolic toolbox:

b = mod(a,m) returns the remainder after division of a by m, where a is the dividend and m is the divisor. From here https://www.mathworks.com/help/matlab/ref/mod.html

Lacking the required toolbox for higher precision numbers, using python in MATLAB might be an option:

p = py.pow(int64(8687205886),int64(5788687615),int64(8369428283))

p = 

  Python long with properties:

    denominator: [1×1 py.long]
           imag: [1×1 py.long]
      numerator: [1×1 py.long]
           real: [1×1 py.long]

    539591274

This is only precision problem, if you accept that means a problem of understanding a floating point arithmetic environment. In MATLAB (as is true in many languages), the number 0.36 is not represented exactly as the fraction 36/100. Instead, MATLAB uses a binary form to represent the number. x = 0.36 It LOOKs like 0.36. But is it? Just like the number 1/3 is NOT representable in a finite number of digits in a decimal form, the fraction 36/100 is not representable in a finite number of binary bits. MATLAB is forced to represent that number as effectively sum(2.^[-2 -4 -5 -6 -11 -13 -17 -18 -19 -20 -22 -24 -25 -26 -31 -33 -37 -38 -39 -40 -42 -44 -45 -46 -51 -53]) which still looks vaguely like the number you wanted. But is it that? In fact, it should be an infinitely repeating number, with extra binary bits down at the botttom, just like 1/3 or 2/3 cannot be represented in decimal form. vpa(sum(sym(2).^[-2 -4 -5 -6 -11 -13 -17 -18 -19 -20 -22 -24 -25 -26 -31 -33 -37 -38 -39 -40 -42 -44 -45 -46 -51 -53]) - sym('0.36')) Now, what happens when you multiply 0.36 with 1e14? It is not an integer. That would only be true if 0.36 was exactly representable. And therefore mod gets in trouble. It does what it can. But you passed it a non-integer, so what do you expect? The same is true of each of your other cases. If you want a valid result, then you could use tools like syms (or my own HPF toolbox) to do all of your work. But expect things to be really incredibly slow. mod(sym('0.36')*1e14,256) My HPF will work, because it stores the numbers in a decimal form, and therefore 0.36 is not stored as an approximation. mod(hpf('0.36')*1e14,256) ans = 0 In the end, you will be far better off understanding floating point arithmetic than to use a powerful tool like that though.

The dyadic transformation seems to be a transformation from [0,1) continuous to [0,1) continuous. I see nothing wrong with your test code if you are trying to implement the dyadic mapping. You should be expecting output in the [0,1)