awkward.fromiter is the one function that was allowed to be written in Python for loops, and hence it is designated to be slow. The function you want for turning regular NumPy arrays into JaggedArrays that happen to have uniform counts is JaggedArray.fromregular. That ought to be considerably faster.
Meanwhile, your original issue is an example of an inconsistency in Awkward 0.x. In Awkward 1.x, the behavior of Awkward Arrays that happen to be regular and NumPy arrays with the same logical meaning are identical. You might want to consider awkward1.from_awkward0 in the awkward1 library to try it out. (It's a separate library because the interface is a little different and I don't want to break anyone's analysis!)
Try:
>>> X[np.ix_(m0, m1)]
array([[ 4, 5, 6],
[ 8, 9, 10]])
From the docs:
Combining multiple Boolean indexing arrays or a Boolean with an integer indexing array can best be understood with the obj.nonzero() analogy. The function ix_ also supports boolean arrays and will work without any surprises.
Another solution (also straight from the docs but less intuitive IMO):
>>> X[m0.nonzero()[0][:, np.newaxis], m1]
array([[ 4, 5, 6],
[ 8, 9, 10]])
The error tells you what you need to do: the mask dimensions need to broadcast together. You can fix this at the source:
m0 = (X>0).all(axis=1, keepdims=True)
m1 = (X<3).any(axis=0, keepdims=True)
>>> X[m0 & m1]
array([ 4, 5, 6, 8, 9, 10])
You only really need to apply keepdims to m0, so you can leave the masks as 1D:
>>> X[m0[:, None] & m1]
array([ 4, 5, 6, 8, 9, 10])
You can reshape to the desired shape:
>>> X[m0[:, None] & m1].reshape(np.count_nonzero(m0), np.count_nonzero(m1))
array([[ 4, 5, 6],
[ 8, 9, 10]])
Another option is to convert the masks to indices:
>>> X[np.flatnonzero(m0)[:, None], np.flatnonzero(m1)]
array([[ 4, 5, 6],
[ 8, 9, 10]])