In [1]: import numpy as np
In [2]: a = np.array([[2,0],[3,0],[3,1],[5,0],[5,1],[5,2]])
In [3]: b = np.zeros((6,3), dtype='int32')
In [4]: b[a[:,0], a[:,1]] = 10
In [5]: b
Out[5]:
array([[ 0, 0, 0],
[ 0, 0, 0],
[10, 0, 0],
[10, 10, 0],
[ 0, 0, 0],
[10, 10, 10]])
Why it works:
If you index b with two numpy arrays in an assignment,
b[x, y] = z
then think of NumPy as moving simultaneously over each element of x and each element of y and each element of z (let's call them xval, yval and zval), and assigning to b[xval, yval] the value zval. When z is a constant, "moving over z just returns the same value each time.
That's what we want, with x being the first column of a and y being the second column of a. Thus, choose x = a[:, 0], and y = a[:, 1].
b[a[:,0], a[:,1]] = 10
Why b[a] = 10 does not work
When you write b[a], think of NumPy as creating a new array by moving over each element of a, (let's call each one idx) and placing in the new array the value of b[idx] at the location of idx in a.
idx is a value in a. So it is an int32. b is of shape (6,3), so b[idx] is a row of b of shape (3,). For example, when idx is
In [37]: a[1,1]
Out[37]: 0
b[a[1,1]] is
In [38]: b[a[1,1]]
Out[38]: array([0, 0, 0])
So
In [33]: b[a].shape
Out[33]: (6, 2, 3)
So let's repeat: NumPy is creating a new array by moving over each element of a and placing in the new array the value of b[idx] at the location of idx in a. As idx moves over a, an array of shape (6,2) would be created. But since b[idx] is itself of shape (3,), at each location in the (6,2)-shaped array, a (3,)-shaped value is being placed. The result is an array of shape (6,2,3).
Now, when you make an assignment like
b[a] = 10
a temporary array of shape (6,2,3) with values b[a] is created, then the assignment is performed. Since 10 is a constant, this assignment places the value 10 at each location in the (6,2,3)-shaped array.
Then the values from the temporary array are reassigned back to b.
See reference to docs. Thus the values in the (6,2,3)-shaped array are copied back to the (6,3)-shaped b array. Values overwrite each other. But the main point is you do not obtain the assignments you desire.
numpy - Use 2d array as list of indices for n-D array - Stack Overflow
python - Indexing a 1D Numpy array using 2D array - Stack Overflow
python - Index 2D numpy array by a 2D array of indices without loops - Stack Overflow
Numpy indices-ifs for 2d array rows?
In [1]: import numpy as np
In [2]: a = np.array([[2,0],[3,0],[3,1],[5,0],[5,1],[5,2]])
In [3]: b = np.zeros((6,3), dtype='int32')
In [4]: b[a[:,0], a[:,1]] = 10
In [5]: b
Out[5]:
array([[ 0, 0, 0],
[ 0, 0, 0],
[10, 0, 0],
[10, 10, 0],
[ 0, 0, 0],
[10, 10, 10]])
Why it works:
If you index b with two numpy arrays in an assignment,
b[x, y] = z
then think of NumPy as moving simultaneously over each element of x and each element of y and each element of z (let's call them xval, yval and zval), and assigning to b[xval, yval] the value zval. When z is a constant, "moving over z just returns the same value each time.
That's what we want, with x being the first column of a and y being the second column of a. Thus, choose x = a[:, 0], and y = a[:, 1].
b[a[:,0], a[:,1]] = 10
Why b[a] = 10 does not work
When you write b[a], think of NumPy as creating a new array by moving over each element of a, (let's call each one idx) and placing in the new array the value of b[idx] at the location of idx in a.
idx is a value in a. So it is an int32. b is of shape (6,3), so b[idx] is a row of b of shape (3,). For example, when idx is
In [37]: a[1,1]
Out[37]: 0
b[a[1,1]] is
In [38]: b[a[1,1]]
Out[38]: array([0, 0, 0])
So
In [33]: b[a].shape
Out[33]: (6, 2, 3)
So let's repeat: NumPy is creating a new array by moving over each element of a and placing in the new array the value of b[idx] at the location of idx in a. As idx moves over a, an array of shape (6,2) would be created. But since b[idx] is itself of shape (3,), at each location in the (6,2)-shaped array, a (3,)-shaped value is being placed. The result is an array of shape (6,2,3).
Now, when you make an assignment like
b[a] = 10
a temporary array of shape (6,2,3) with values b[a] is created, then the assignment is performed. Since 10 is a constant, this assignment places the value 10 at each location in the (6,2,3)-shaped array.
Then the values from the temporary array are reassigned back to b.
See reference to docs. Thus the values in the (6,2,3)-shaped array are copied back to the (6,3)-shaped b array. Values overwrite each other. But the main point is you do not obtain the assignments you desire.
TL;DR: Use advanced indexing: b[*a.T] = 10
You can also transpose the index array a, convert the result into a tuple and index the array b and assign a value. Converting the index array into a tuple (or unpacking it inside a []) ensures that multidimensional indexing works as expected. This is assignment by advanced indexing.
a = np.array([[2, 0], [3, 0], [3, 1], [5, 0], [5, 1], [5, 2]])
b = np.zeros((6,3), dtype ='int32')
b[*a.T] = 10
# or
b[tuple(a.T)] = 10
# or
b[(*a.T,)] = 10
# or
b[(*a.T.tolist(),)] = 10
All of them produce the expected output of
array([[ 0, 0, 0],
[ 0, 0, 0],
[10, 0, 0],
[10, 10, 0],
[ 0, 0, 0],
[10, 10, 10]])
Here's a 2d example:
In [1]: A=np.arange(10,22).reshape(3,4)
In [2]: A
Out[2]:
array([[10, 11, 12, 13],
[14, 15, 16, 17],
[18, 19, 20, 21]])
In [3]: ind=np.array([[0,1],[1,3],[2,0],[0,2]])
In [4]: ind
Out[4]:
array([[0, 1],
[1, 3],
[2, 0],
[0, 2]])
In [5]: A[ind[:,0],ind[:,1]]
Out[5]: array([11, 17, 18, 12])
or for your variables,
A[indices[:,0], indices[:,1], indices[:,2]]
Or more generally:
In [8]: tuple(ind.T)
Out[8]: (array([0, 1, 2, 0]), array([1, 3, 0, 2]))
In [9]: A[tuple(ind.T)]
Out[9]: array([11, 17, 18, 12])
This is based on the idea that A[a,b] is the same as A[(a,b)]. And when a and b are matching lists or arrays, it selects values by pairing them up, roughly the same as
[A[i,j] for i,j in zip(a,b)]
For a product like indexing, the index arrays need to have more dimensions. ix_ is a handy way of generating such arrays:
In [53]: np.ix_(ind[:,0],ind[:,1])
Out[53]:
(array([[0],
[1],
[2],
[0]]), array([[1, 3, 0, 2]]))
In [54]: A[np.ix_(ind[:,0],ind[:,1])]
Out[54]:
array([[11, 13, 10, 12],
[15, 17, 14, 16],
[19, 21, 18, 20],
[11, 13, 10, 12]])
In [56]: A[ind[:,[0]],ind[:,1]]
Out[56]:
array([[11, 13, 10, 12],
[15, 17, 14, 16],
[19, 21, 18, 20],
[11, 13, 10, 12]])
You could use np.ravel_multi_index to generate the linear indices and then extract the selective elements from A with linear-indexing using np.take like so -
np.take(A,np.ravel_multi_index(indices.T,A.shape))
When using arrays of indices to index another array, the shape of each index array should match the shape of the output array. You want the column indices to match inds, and you want the row indices to match the row of the output, something like:
array([[0, 0],
[1, 1],
[2, 2]])
You can just use a single column of the above, due to broadcasting, so you can use np.arange(3)[:,None] is the vertical arange because None inserts a new axis:
>>> np.arange(3)[:, None]
array([[0],
[1],
[2]])
Finally, together:
>>> a[np.arange(3)[:,None], inds]
array([[0, 3], # a[0,[0,1]]
[6, 0], # a[1,[1,2]]
[0, 9]]) # a[2,[0,2]]
It’s possible, although somewhat non-obvious to do this as follows:
>>> a[np.arange(a.shape[0])[:, None], inds]
array([[0, 3],
[6, 0],
[0, 9]])
The index np.arange(a.shape[0]) simply indexes the rows to which the array of column indices inds applies. Appending [:, None] modifies the shape of this array such that its shape is (a.shape[0], 1), i.e. each row index is in a separate row of a 1-column-wide 2D array.
The basic principle is that the number of dimensions in the index arrays must agree, and their shapes must also do so. See documentation for np.ix_ to get a feel for this.
hello,
there is a feature in numpy that lets you find array elements with certain properties, for example:
import numpy as np arr = np.array([5,6,2,6,7,9,2,1,4,7,0,6]) print(arr[arr>5]) => [6, 6, 7, 9, 7, 6] # this works great for quick substitutions: arr2 = np.array([9,9,9,9,9,9,9,9,9,9,9,9]) arr[arr<5] = arr2[arr<5] print(arr) => [5, 6, 9, 6, 7, 9, 9, 9, 9, 7, 9, 6] # it even keeps dimensions: arr2d = np.array([[0,1,2],[3,4,5],[6,7,8]]) arr2d[arr2d<4] = 99 print(arr2d) =>[[99 99 99] [99 4 5] [ 6 7 8]]
Now to my question:
The 'if thingies' loop through every element of an array regardless of it's dimensions, creates a 1d bool array, which is then fed into the square brackets and then the operation is only performed if the bool at the current index is true.
Is there any way that this 'if' does not look at each individual element, but for example at a whole row? Can you specify how "deep" the search should be?
# task (not a real task, just demonstration): replace all rows that have a sum > 10 with a row containing only twos # replace? yes yes no yes no arr = np.array([[2,3,7], [9,1,1], [3,4,1], [4, 4, 4], [0, 7, 1]]) # this search now should only penetrate the first layer of the 2d array and not the second one, so i dont get individual numbers in my comparison/selection, but whole "rows/subarrays" arr[*black magic*] = (2, 2, 2) print(arr) => [[2,2,2], [2,2,2], [3,4,1], [2, 2, 2], [0, 7, 1]]
Thank you for your help!
I know this is very confusing and probably even more confusing if the person reading this is not me, so please tell me if what i wrote is utter nonsense
i know that you can do it with for loops or list comprehension, but i want to know if this specific form works, because i think it's a very cool feature
You can use choose to make the selection:
>>> z_indices.choose(val_arr)
array([[ 9, 1, 20],
[ 3, 4, 14],
[24, 7, 17]])
The function choose is incredibly useful, but can be somewhat tricky to make sense of. Essentially, given an array (val_arr) we can make a series of choices (z_indices) from each n-dimensional slice along the first axis.
Also: any fancy indexing operation will create a new array rather than a view of the original data. It is not possible to index val_arr with z_indices without creating a brand new array.
With readability, np.choose definitely looks great.
If performance is of essence, you can calculate the linear indices and then use np.take or use a flattened version with .ravel() and extract those specific elements from val_arr. The implementation would look something like this -
def linidx_take(val_arr,z_indices):
# Get number of columns and rows in values array
_,nC,nR = val_arr.shape
# Get linear indices and thus extract elements with np.take
idx = nC*nR*z_indices + nR*np.arange(nR)[:,None] + np.arange(nC)
return np.take(val_arr,idx) # Or val_arr.ravel()[idx]
Runtime tests and verify results -
Ogrid based solution from here is made into a generic version for these tests, like so :
In [182]: def ogrid_based(val_arr,z_indices):
...: v_shp = val_arr.shape
...: y,x = np.ogrid[0:v_shp[1], 0:v_shp[2]]
...: return val_arr[z_indices, y, x]
...:
Case #1: Smaller datasize
In [183]: val_arr = np.random.rand(30,30,30)
...: z_indices = np.random.randint(0,30,(30,30))
...:
In [184]: np.allclose(z_indices.choose(val_arr),ogrid_based(val_arr,z_indices))
Out[184]: True
In [185]: np.allclose(z_indices.choose(val_arr),linidx_take(val_arr,z_indices))
Out[185]: True
In [187]: %timeit z_indices.choose(val_arr)
1000 loops, best of 3: 230 µs per loop
In [188]: %timeit ogrid_based(val_arr,z_indices)
10000 loops, best of 3: 54.1 µs per loop
In [189]: %timeit linidx_take(val_arr,z_indices)
10000 loops, best of 3: 30.3 µs per loop
Case #2: Bigger datasize
In [191]: val_arr = np.random.rand(300,300,300)
...: z_indices = np.random.randint(0,300,(300,300))
...:
In [192]: z_indices.choose(val_arr) # Seems like there is some limitation here with bigger arrays.
Traceback (most recent call last):
File "<ipython-input-192-10c3bb600361>", line 1, in <module>
z_indices.choose(val_arr)
ValueError: Need between 2 and (32) array objects (inclusive).
In [194]: np.allclose(linidx_take(val_arr,z_indices),ogrid_based(val_arr,z_indices))
Out[194]: True
In [195]: %timeit ogrid_based(val_arr,z_indices)
100 loops, best of 3: 3.67 ms per loop
In [196]: %timeit linidx_take(val_arr,z_indices)
100 loops, best of 3: 2.04 ms per loop