The nice thing about array slicing in numpy is you don't need the for loops that you are using. Also the reason that it is only putting the center element is because you only put a single element there (c1[c1mid,c1mid] is a single number) here is what you could do:
z[7:12,7:12] = c1
z[7:12,27:32] = c2
z[26:33,6:14] = c3
z[25:34,25:33] = c4
Answer from jfish003 on Stack OverflowThe nice thing about array slicing in numpy is you don't need the for loops that you are using. Also the reason that it is only putting the center element is because you only put a single element there (c1[c1mid,c1mid] is a single number) here is what you could do:
z[7:12,7:12] = c1
z[7:12,27:32] = c2
z[26:33,6:14] = c3
z[25:34,25:33] = c4
The inbuilt np.ix_ works perfectly
import numpy as np
a=np.zeros([5,5])
b=np.random.rand(3,3)*100
idx=[0,2,3]
print(a)
a[np.ix_(idx, idx)]+=b
print(a)
the output is
[[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]]
[[56.22112929 0. 57.43572879 2.90797715 0. ]
[ 0. 0. 0. 0. 0. ]
[54.08128804 0. 23.53431307 24.03463619 0. ]
[96.7227866 0. 3.01937951 68.09775321 0. ]
[ 0. 0. 0. 0. 0. ]]
python - Numpy submatrix operations - Stack Overflow
python - Using numpy to Obtain Submatrix - Stack Overflow
python - How to assign to square submatrices in big matrix without loops in numpy - Stack Overflow
matrix - python: how to create submatrices? Numpy - Stack Overflow
There are several ways to get submatrix in numpy:
In [35]: ri = [0,2]
...: ci = [2,3]
...: a[np.reshape(ri, (-1, 1)), ci]
Out[35]:
array([[ 2, 3],
[10, 11]])
In [36]: a[np.ix_(ri, ci)]
Out[36]:
array([[ 2, 3],
[10, 11]])
In [37]: s=a[np.ix_(ri, ci)]
In [38]: np.may_share_memory(a, s)
Out[38]: False
note that the submatrix you get is a new copy, not a view of the original mat.
You only need to makes cols and rows be a numpy array, and then you can just use the [] as:
import numpy as np
a = np.array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
cols = np.array([True, False, True])
rows = np.array([False, False, True, True])
result = a[cols][:,rows]
print(result)
print(type(result))
# [[ 2 3]
# [10 11]]
# <class 'numpy.ndarray'>
It seems to me that you're trying to do a simple convolution?
def do(m):
rows, cols = m.shape
newMatrix = np.zeros_like(m)
for i in range(1, rows-1):
for j in range(1, cols-1):
sub = matrix[i-1:i+2, j-1:j+2]
newMatrix[i][j] = numpy.sum(xWeight * sub)
return newMatrix[1:-1, 1:-1]
>>> res1 = do(matrix)
>>> res2 = scipy.signal.convolve2d(matrix, xWeight)[2:-2,2:-2]
>>> np.allclose(np.abs(res1), np.abs(res2))
True
Didn't went into details about the sign, but that should hopefully put you on the right track.
I found a solution in numpy.lib.stride_tricks
from numpy.lib.stride_tricks import as_strided
In the method:
expansion = stride.as_strided(matrix, shape = (numRows-2, numCols-2, 3, 3), strides = matrix.strides * 2)
xWeight = numpy.array([[-1./8, 0, 1./8], [-2./8, 0, 2./8], [-1./8, 0, 1./8]])
yWeight = numpy.array([[1./8, 2./8, 1./8], [0, 0, 0], [-1./8, -2./8, -1./8]])
dx = xWeight * expansion
dy = yWeight * expansion
dx = numpy.sum(numpy.sum(dx, axis=3), axis=2)
dy = numpy.sum(numpy.sum(dy, axis=3), axis=2)
There may well be a better solution, but this is sufficiently simple and general purpose for what I was after. This went through a 1600x1200 matrix in 3.41 seconds, vs 188.47 seconds using for loops.
(Feel free to offer said better solution, if you have it)
Here is how you can do it:
>>> A[3:5, 3:5] = B
>>> A
array([[ 1. , 1. , 1. , 1. , 1. ],
[ 1. , 1. , 1. , 1. , 1. ],
[ 1. , 1. , 1. , 1. , 1. ],
[ 1. , 1. , 1. , 0.1, 0.2],
[ 1. , 1. , 1. , 0.3, 0.4]])
In general, for example, for non-contiguous rows/cols
use numpy.putmask(a, mask, values) (Sets a.flat[n] = values[n] for each n where mask.flat[n]==True)
For example
In [1]: a = np.zeros((3, 3))
Out [1]: a
array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])
In [2]: values = np.ones((2, 2))
Out [2]: values
array([[1., 1.],
[1., 1.]])
In [3]: mask = np.zeros((3, 3), dtype=bool)
In [4]: mask[0,0] = mask[0,1] = mask[1,1] = mask[2,2] = True
Out [4]: mask
array([[ True, True, False],
[False, True, False],
[False, False, True]])
In [5] np.putmask(a, mask, values)
Out [5] a
array([[1., 1., 0.],
[0., 1., 0.],
[0., 0., 1.]])
Is n potentially large, so the result is a large sparse matrix with nonzero values concentrated along the diagonal? Sparse matrices are designed with this kind of matrix in mind (from FD and FE PDE problems). I did this a lot in MATLAB, and some with the scipy sparse module.
That module has a block definition mode that might work, but what I'm more familiar with is the coo to csr route.
In the coo format, nonzero elements are defined by 3 vectors, i, j, and data. You can collect all the values for A, B, etc in these arrays (applying the appropriate offset for the values in B etc), without worrying about overlaps. Then when that format is converted to csr (for matrix calculations) the overlapping values are summed - which is exactly what you want.
I think the sparse documentation has some simple examples of this. Conceptually the simplest thing to do is iterate over the n submatrices, and collect the values in those 3 arrays. But I also worked out a more complex system whereby it can be done as one big array operation, or by iterating over a smaller dimension. For example each submatrix has 16 values. In a realistic case 16 will be much smaller than n.
I'd have play around with code to give a more concrete example.
==========================
Here's a simple example with 3 blocks - functional, but not the most efficient
Define 3 blocks:
In [620]: A=np.ones((4,4),int)
In [621]: B=np.ones((4,4),int)*2
In [622]: C=np.ones((4,4),int)*3
lists to collect values in; could be arrays, but it is easy, and relatively efficient to append or extend lists:
In [623]: i, j, dat = [], [], []
In [629]: def foo(A,n):
# turn A into a sparse, and add it's points to the arrays
# with an offset of 'n'
ac = sparse.coo_matrix(A)
i.extend(ac.row+n)
j.extend(ac.col+n)
dat.extend(ac.data)
In [630]: foo(A,0)
In [631]: i
Out[631]: [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3]
In [632]: j
Out[632]: [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]
In [633]: foo(B,1)
In [634]: foo(C,2) # do this in a loop in the real world
In [636]: M = sparse.csr_matrix((dat,(i,j)))
In [637]: M
Out[637]:
<6x6 sparse matrix of type '<class 'numpy.int32'>'
with 30 stored elements in Compressed Sparse Row format>
In [638]: M.A
Out[638]:
array([[1, 1, 1, 1, 0, 0],
[1, 3, 3, 3, 2, 0],
[1, 3, 6, 6, 5, 3],
[1, 3, 6, 6, 5, 3],
[0, 2, 5, 5, 5, 3],
[0, 0, 3, 3, 3, 3]], dtype=int32)
If I've done this right, overlapping values of A,B,C are summed.
More generally:
In [21]: def foo1(mats):
i,j,dat = [],[],[]
for n,mat in enumerate(mats):
A = sparse.coo_matrix(mat)
i.extend(A.row+n)
j.extend(A.col+n)
dat.extend(A.data)
M = sparse.csr_matrix((dat,(i,j)))
return M
....:
In [22]: foo1((A,B,C,B,A)).A
Out[22]:
array([[1, 1, 1, 1, 0, 0, 0, 0],
[1, 3, 3, 3, 2, 0, 0, 0],
[1, 3, 6, 6, 5, 3, 0, 0],
[1, 3, 6, 8, 7, 5, 2, 0],
[0, 2, 5, 7, 8, 6, 3, 1],
[0, 0, 3, 5, 6, 6, 3, 1],
[0, 0, 0, 2, 3, 3, 3, 1],
[0, 0, 0, 0, 1, 1, 1, 1]], dtype=int32)
Coming up with a way of doing this more efficiently may depend on how the individual submatrices are generated. If they are created iteratively, you might as well collect the i,j,data values iteratively as well.
==========================
Since the submatrices are dense, we can get the appropriate i,j,data values directly, without going through a coo intermediary. And without looping if the A,B,C are collected into one larger array.
If I modify foo1 to return a coo matrix, I see the i,j,data lists (as arrays) as given, without summation of duplicates. In the example with 5 matrices, I get 80 element arrays, which can be reshaped as
In [110]: f.col.reshape(-1,16)
Out[110]:
array([[0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3],
[1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4],
[2, 3, 4, 5, 2, 3, 4, 5, 2, 3, 4, 5, 2, 3, 4, 5],
[3, 4, 5, 6, 3, 4, 5, 6, 3, 4, 5, 6, 3, 4, 5, 6],
[4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7]], dtype=int32)
In [111]: f.row.reshape(-1,16)
Out[111]:
array([[0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3],
[1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4],
[2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5],
[3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6],
[4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7]], dtype=int32)
In [112]: f.data.reshape(-1,16)
Out[112]:
array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])
I should be able generate those without a loop, especially the row and col.
In [143]: mats=[A,B,C,B,A]
the coordinates for the elements of an array
In [144]: I,J=[i.ravel() for i in np.mgrid[range(A.shape[0]),range(A.shape[1])]]
replicate them with offset via broadcasting
In [145]: x=np.arange(len(mats))[:,None]
In [146]: I=I+x
In [147]: J=J+x
Collect the data into one large array:
In [148]: D=np.concatenate(mats,axis=0)
In [149]: f=sparse.csr_matrix((D.ravel(),(I.ravel(),J.ravel())))
or as a compact function
def foo3(mats):
A = mats[0]
n,m = A.shape
I,J = np.mgrid[range(n), range(m)]
x = np.arange(len(mats))[:,None]
I = I.ravel()+x
J = J.ravel()+x
D=np.concatenate(mats,axis=0)
f=sparse.csr_matrix((D.ravel(),(I.ravel(),J.ravel())))
return f
In this modest example the 2nd version is 2x faster; the first scales linearly with the length of the list; the 2nd is almost independent of its length.
In [158]: timeit foo1(mats)
1000 loops, best of 3: 1.3 ms per loop
In [160]: timeit foo3(mats)
1000 loops, best of 3: 653 µs per loop
The simple for-loop way would be to add each 4x4 matrix to an appropriate slice of the big zero matrix:
for i, small_m in enumerate(small_matrices):
big_m[i:i+4, i:i+4] += small_m
You could also do this with no Python loops by creating a strided view of the zero matrix and using np.add.at for unbuffered addition. This should be particularly efficient if your 4x4 matrices are packed into a k-by-4-by-4 array:
import numpy as np
from numpy.lib.stride_tricks import as_strided
# Create a view of big_m with the shape of small_matrices.
# strided_view[i] is a view of big_m[i:i+4, i:i+4]
strides = (sum(big_m.strides),) + big_m.strides
strided_view = as_strided(big_m, shape=small_matrices.shape, strides=strides)
np.add.at(strided_view, np.arange(small_matrices.shape[0]), small_matrices)
Hi,
I have an MxN matrix (list of lists), where each element is a tuple of 2 ints.
Given a rectangle size PxQ, where P<=M, Q<=N), I need to find the submatrix inside the MxN matrix which, when calculating the sum of the second element of each tuple inside the rectangle, returns the highest result which is not larger than a number B. Each submatrix is defined by its upper-left corner.
For example, the MxN matrix can be:
[ [(1, 2), (1, 1), (0, 3), (4, 0)],
[(10, 10), (5, 7), (1, 3), (9, 2)],
[(0, 0), (1, 9), (0, 0), (1, 1)] ]
and the rectangle size can be 2x3, so there are 4 submatrices to go over:
[(1, 2), (1, 1), (0, 3)
(10, 10), (5, 7), (1, 3)]
[(1, 1), (0, 3), (4, 0)
(5, 7), (1, 3), (9, 2)]
[(10, 10), (5, 7), (1, 3)
(0, 0), (1, 9), (0, 0)]
[(5, 7), (1, 3), (9, 2)
(1, 9), (0, 0), (1, 1)]
If B=27, the correct submatrix, in this case, is the second one, since it has the highest sum of 2nd elements, which is 2+1+3+10+7+3=26, which is smaller than B. The third submatrix yields a larger sum (29) but 29 > 27 so it is not the right answer.
I'm looking for an efficient way to go over the submatrices and determine if the sum of the 2nd elements is the largest. Is there a faster way than using for loops?
If you want to add B to A with the upper left-hand corner of B going to index (r, c) in A, you can do it using the index and the shape attribute of B:
A[r:r+B.shape[0], c:c+B.shape[1]] += B
If you want to just set the elements (overwrite instead of adding), replace += with =. In your particular example:
>>> A = np.zeros((5, 6), dtype=int)
>>> B = np.r_[np.arange(2, 10), 3].reshape(3, 3)
>>> r, c = 1, 2
>>> A[r:r+B.shape[0], c:c+B.shape[1]] += B
>>> A
array([[0, 0, 0, 0, 0, 0],
[0, 0, 2, 3, 4, 0],
[0, 0, 5, 6, 7, 0],
[0, 0, 8, 9, 3, 0],
[0, 0, 0, 0, 0, 0]])
The indexing operation produces a view into A since it is simple indexing, meaning that the data is not copied, which makes the operation fairly efficient for large arrays.
You can pad the b array into the same shape with a. numpy.pad
import numpy as np
a = np.array([[0,0,0,0,0,0],
[0,0,0,0,0,0],
[0,0,0,0,0,0],
[0,0,0,0,0,0],
[0,0,0,0,0,0]])
b = np.array([[2,3,4],
[5,6,7],
[8,9,3]])
b = np.pad(b, ((1,1) , (2,1)), mode = 'constant', constant_values=(0, 0))
print(a+b)
After padding b will be
[[0 0 0 0 0 0]
[0 0 2 3 4 0]
[0 0 5 6 7 0]
[0 0 8 9 3 0]
[0 0 0 0 0 0]]
a+b will be
[[0 0 0 0 0 0]
[0 0 2 3 4 0]
[0 0 5 6 7 0]
[0 0 8 9 3 0]
[0 0 0 0 0 0]]
The ((1,1) , (2,1)) means you add 1 row on top, one row on bottom, 2 columns on left, 1 columns on right. All added row and columns are zeros because of mode = 'constant', constant_values=(0, 0).
So you can input the index you want to add the matrix
It probably is more useful to work with the transpose of W rather than W itself, both for human-readability and to facilitate writing the code. This means that the entries that affect each S_i are grouped together in one of the inner parentheses of W, i.e. in a row of W rather than a column as you have it now.
Then, S_i = np.array[S[j,:] for j in np.shape(S)[0] if W_T[i,j] == 1], where W_T is the transpose of W. If you need/want to stick with W as is, you need to reverse the indices i and j.
As for the outer loop, you could try to nest this in another similar comprehension without an if statement--however this might be awkward since you aren't actually building one output matrix (the S_i can easily be different dimensions, unless you're somehow guaranteed to have the same number of 1s in every column of W). This in fact raises the question of what you want--a list of these arrays S_i? Otherwise if they are separate variables as you have it written, there's no good way to refer to them in a generalizable way as they don't have indices.
Numpy can do this directly.
import numpy as np
S = np.array([[1,1],[1,2],[1,3],[1,4],[1,5]])
W = np.array([[1,0,0],[1,1,0],[1,1,1],[0,1,1],[0,0,1]])
for row in range(W.shape[1]):
print(S[W[:,row]==1])
Output:
[[1 1]
[1 2]
[1 3]]
[[1 2]
[1 3]
[1 4]]
[[1 3]
[1 4]
[1 5]]