Give np.ix_ a try:
Y[np.ix_([0,3],[0,3])]
This returns your desired result:
In [25]: Y = np.arange(16).reshape(4,4)
In [26]: Y[np.ix_([0,3],[0,3])]
Out[26]:
array([[ 0, 3],
[12, 15]])
Give np.ix_ a try:
Y[np.ix_([0,3],[0,3])]
This returns your desired result:
In [25]: Y = np.arange(16).reshape(4,4)
In [26]: Y[np.ix_([0,3],[0,3])]
Out[26]:
array([[ 0, 3],
[12, 15]])
One solution is to index the rows/columns by slicing/striding. Here's an example where you are extracting every third column/row from the first to last columns (i.e. the first and fourth columns)
In [1]: import numpy as np
In [2]: Y = np.arange(16).reshape(4, 4)
In [3]: Y[0:4:3, 0:4:3]
Out[1]: array([[ 0, 3],
[12, 15]])
This gives you the output you were looking for.
For more info, check out this page on indexing in NumPy.
How to go over submatrices of a matrix - and fast?
python - Using numpy to Obtain Submatrix - Stack Overflow
python - Numpy submatrix operations - Stack Overflow
a) Submatrix Extraction with NumPy Write a Python function extract_submatrix (matrix, rows_to_remove, cols_to_remove). This function takes the following parameters: - matrix: A 2D NumPy array (matrix) of shape (M, N) ( M,N>1). - rows_to_remove: A list of row indices to be removed from the original matrix. - cols_to_remove: A list of column indices to be
Hi,
I have an MxN matrix (list of lists), where each element is a tuple of 2 ints.
Given a rectangle size PxQ, where P<=M, Q<=N), I need to find the submatrix inside the MxN matrix which, when calculating the sum of the second element of each tuple inside the rectangle, returns the highest result which is not larger than a number B. Each submatrix is defined by its upper-left corner.
For example, the MxN matrix can be:
[ [(1, 2), (1, 1), (0, 3), (4, 0)],
[(10, 10), (5, 7), (1, 3), (9, 2)],
[(0, 0), (1, 9), (0, 0), (1, 1)] ]
and the rectangle size can be 2x3, so there are 4 submatrices to go over:
[(1, 2), (1, 1), (0, 3)
(10, 10), (5, 7), (1, 3)]
[(1, 1), (0, 3), (4, 0)
(5, 7), (1, 3), (9, 2)]
[(10, 10), (5, 7), (1, 3)
(0, 0), (1, 9), (0, 0)]
[(5, 7), (1, 3), (9, 2)
(1, 9), (0, 0), (1, 1)]
If B=27, the correct submatrix, in this case, is the second one, since it has the highest sum of 2nd elements, which is 2+1+3+10+7+3=26, which is smaller than B. The third submatrix yields a larger sum (29) but 29 > 27 so it is not the right answer.
I'm looking for an efficient way to go over the submatrices and determine if the sum of the 2nd elements is the largest. Is there a faster way than using for loops?
It seems to me that you're trying to do a simple convolution?
def do(m):
rows, cols = m.shape
newMatrix = np.zeros_like(m)
for i in range(1, rows-1):
for j in range(1, cols-1):
sub = matrix[i-1:i+2, j-1:j+2]
newMatrix[i][j] = numpy.sum(xWeight * sub)
return newMatrix[1:-1, 1:-1]
>>> res1 = do(matrix)
>>> res2 = scipy.signal.convolve2d(matrix, xWeight)[2:-2,2:-2]
>>> np.allclose(np.abs(res1), np.abs(res2))
True
Didn't went into details about the sign, but that should hopefully put you on the right track.
I found a solution in numpy.lib.stride_tricks
from numpy.lib.stride_tricks import as_strided
In the method:
expansion = stride.as_strided(matrix, shape = (numRows-2, numCols-2, 3, 3), strides = matrix.strides * 2)
xWeight = numpy.array([[-1./8, 0, 1./8], [-2./8, 0, 2./8], [-1./8, 0, 1./8]])
yWeight = numpy.array([[1./8, 2./8, 1./8], [0, 0, 0], [-1./8, -2./8, -1./8]])
dx = xWeight * expansion
dy = yWeight * expansion
dx = numpy.sum(numpy.sum(dx, axis=3), axis=2)
dy = numpy.sum(numpy.sum(dy, axis=3), axis=2)
There may well be a better solution, but this is sufficiently simple and general purpose for what I was after. This went through a 1600x1200 matrix in 3.41 seconds, vs 188.47 seconds using for loops.
(Feel free to offer said better solution, if you have it)
There are several ways to get submatrix in numpy:
In [35]: ri = [0,2]
...: ci = [2,3]
...: a[np.reshape(ri, (-1, 1)), ci]
Out[35]:
array([[ 2, 3],
[10, 11]])
In [36]: a[np.ix_(ri, ci)]
Out[36]:
array([[ 2, 3],
[10, 11]])
In [37]: s=a[np.ix_(ri, ci)]
In [38]: np.may_share_memory(a, s)
Out[38]: False
note that the submatrix you get is a new copy, not a view of the original mat.
You only need to makes cols and rows be a numpy array, and then you can just use the [] as:
import numpy as np
a = np.array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
cols = np.array([True, False, True])
rows = np.array([False, False, True, True])
result = a[cols][:,rows]
print(result)
print(type(result))
# [[ 2 3]
# [10 11]]
# <class 'numpy.ndarray'>
To optimize the idea would be to minimize the computations once we are inside the loop. So, with that in mind, we would rearrange the rows of the array, sorted by the first column. Then, get the indices that define the boundaries. Finally, start our loop and simply slice for each group to get a submatrix at each iteration. Slicing is virtually free when working with arrays, so that should help us.
Thus, one implementation would be -
a0 = data_mat[:,0]
sidx = a0.argsort()
sd = data_mat[sidx] # sorted data_mat
idx = np.flatnonzero(np.concatenate(( [True], sd[1:,0] != sd[:-1,0], [True] )))
for i,j in zip(idx[:-1], idx[1:]):
tmp_mat = sd[i:j]
print tmp_mat
If you are looking to store each submatrix as an array to have a list of arrays as the final output, simply do -
[sd[i:j] for i,j in zip(idx[:-1], idx[1:])]
For sorted data_mat
For a case with data_mat already being sorted as shown in the sample, we could avoid sorting the entire array and directly use the first column, like so -
a0 = data_mat[:,0]
idx = np.flatnonzero(np.concatenate(( [True], a0[1:] != a0[:-1], [True] )))
for i,j in zip(idx[:-1], idx[1:]):
tmp_mat = data_mat[i:j]
print(tmp_mat)
Again, to get all those submatrices as a list of arrays, use -
[data_mat[i:j] for i,j in zip(idx[:-1], idx[1:])]
Note that the submatrices that we would get with this one would be in a different order than with the sorting done in the previous approach.
Benchmarking for sorted data_mat
Approaches -
# @Daniel F's soln-2
def split_app(data_mat):
idx = np.flatnonzero(data_mat[1:, 0] != data_mat[:-1, 0]) + 1
return np.split(data_mat, idx)
# Proposed in this post
def zip_app(data_mat):
a0 = data_mat[:,0]
idx = np.flatnonzero(np.concatenate(( [True], a0[1:] != a0[:-1], [True] )))
return [data_mat[i:j] for i,j in zip(idx[:-1], idx[1:])]
Timings -
In the sample we had a submatrix of max length 6. So, let's extend to a bigger case keeping it with the same pattern -
In [442]: a = np.random.randint(0,100000,(6*100000,4)); a[:,0].sort()
In [443]: %timeit split_app(a)
10 loops, best of 3: 88.8 ms per loop
In [444]: %timeit zip_app(a)
10 loops, best of 3: 40.2 ms per loop
In [445]: a = np.random.randint(0,1000000,(6*1000000,4)); a[:,0].sort()
In [446]: %timeit split_app(a)
1 loop, best of 3: 917 ms per loop
In [447]: %timeit zip_app(a)
1 loop, best of 3: 414 ms per loop
You can do this with boolean indexing.
unique_ids = np.unique(data_mat[:, 0])
masks = np.equal.outer(unique_ids, data_mat[:, 0])
for mask in masks:
tmp_mat = data_mat[mask]
# do something with tmp_mat ...
print(tmp_mat)
To answer this question, we have to look at how indexing a multidimensional array works in Numpy. Let's first say you have the array x from your question. The buffer assigned to x will contain 16 ascending integers from 0 to 15. If you access one element, say x[i,j], NumPy has to figure out the memory location of this element relative to the beginning of the buffer. This is done by calculating in effect i*x.shape[1]+j (and multiplying with the size of an int to get an actual memory offset).
If you extract a subarray by basic slicing like y = x[0:2,0:2], the resulting object will share the underlying buffer with x. But what happens if you acces y[i,j]? NumPy can't use i*y.shape[1]+j to calculate the offset into the array, because the data belonging to y is not consecutive in memory.
NumPy solves this problem by introducing strides. When calculating the memory offset for accessing x[i,j], what is actually calculated is i*x.strides[0]+j*x.strides[1] (and this already includes the factor for the size of an int):
x.strides
(16, 4)
When y is extracted like above, NumPy does not create a new buffer, but it does create a new array object referencing the same buffer (otherwise y would just be equal to x.) The new array object will have a different shape then x and maybe a different starting offset into the buffer, but will share the strides with x (in this case at least):
y.shape
(2,2)
y.strides
(16, 4)
This way, computing the memory offset for y[i,j] will yield the correct result.
But what should NumPy do for something like z=x[[1,3]]? The strides mechanism won't allow correct indexing if the original buffer is used for z. NumPy theoretically could add some more sophisticated mechanism than the strides, but this would make element access relatively expensive, somehow defying the whole idea of an array. In addition, a view wouldn't be a really lightweight object anymore.
This is covered in depth in the NumPy documentation on indexing.
Oh, and nearly forgot about your actual question: Here is how to make the indexing with multiple lists work as expected:
x[[[1],[3]],[1,3]]
This is because the index arrays are broadcasted to a common shape. Of course, for this particular example, you can also make do with basic slicing:
x[1::2, 1::2]
As Sven mentioned, x[[[0],[2]],[1,3]] will give back the 0 and 2 rows that match with the 1 and 3 columns while x[[0,2],[1,3]] will return the values x[0,1] and x[2,3] in an array.
There is a helpful function for doing the first example I gave, numpy.ix_. You can do the same thing as my first example with x[numpy.ix_([0,2],[1,3])]. This can save you from having to enter in all of those extra brackets.