You can use boolean indexing.
In [413]: x[x[:, -1] != 0]
Out[413]:
array([[0, 2, 1],
[0, 1, 1],
[1, 0, 2],
[2, 1, 2]])
x[:, -1]will retrieve the last columnx[:, -1] != 0returns a boolean maskUse the mask to index into the original array
I have a 2d numpy array M, and some 1d array a, say M = [[1,2], [3,4],[5,6]] and a =[1, 3]. I want to remove the rows of M that have the same element as a in a colom. Thus in this case I would want to remove [1,2]. Thanks
python - Deleting row in numpy array based on condition - Stack Overflow
python - How do I remove rows from a numpy array based on multiple conditions? - Stack Overflow
python - deleting rows in numpy array - Stack Overflow
numpy - Python delete a row that meets a condition - Stack Overflow
If you want to keep using numpy, the solution isn't hard.
data = data[np.logical_not(np.logical_and(data[:,0] > 20, data[:,0] < 25))]
data = data[np.logical_not(np.logical_and(data[:,0] > 30, data[:,0] < 35))]
Or if you want to combine it all into one statement,
data = data[
np.logical_not(np.logical_or(
np.logical_and(data[:,0] > 20, data[:,0] < 25),
np.logical_and(data[:,0] > 30, data[:,0] < 35)
))
]
To explain, conditional statements like data[:,0] < 25 create boolean arrays that track, element-by-element, where the condition in an array is true or false. In this case, it tells you where the first column of data is less than 25.
You can also index numpy arrays with these boolean arrays. A statement like data[data[:,0] > 30] extracts all the rows where data[:,0] > 30 is true, or all the rows where the first element is greater than 30. This kind of conditional indexing is how you extract the rows (or columns, or elements) that you want.
Finally, we need logical tools to combine boolean arrays element-by-element. Regular and, or, and not statements don't work because they try to combine the boolean arrays together as a whole. Fortunately, numpy provides a set of these tools for use in the form of np.logical_and, np.logical_or, and np.logical_not. With these, we can combine our boolean arrays element-wise to find rows that satisfy more complicated conditions.
Find below my solution to the problem of deletion specific rows from a numpy array. The solution is provided as one-liner of the form:
# Remove the rows whose first item is between 20 and 25
A = np.delete(A, np.where( np.bitwise_and( (A[:,0]>=20), (A[:,0]<=25) ) )[0], 0)
and is based on pure numpy functions (np.bitwise_and, np.where, np.delete).
A = np.array( [ [ 18, 6.215, 0.025 ],
[ 19, 6.203, 0.025 ],
[ 20, 6.200, 0.025 ],
[ 21, 6.205, 0.025 ],
[ 22, 6.201, 0.026 ],
[ 23, 6.197, 0.026 ],
[ 24, 6.188, 0.024 ],
[ 25, 6.187, 0.023 ],
[ 26, 6.189, 0.021 ],
[ 27, 6.188, 0.020 ],
[ 28, 6.192, 0.019 ],
[ 29, 6.185, 0.020 ],
[ 30, 6.189, 0.019 ],
[ 31, 6.191, 0.018 ],
[ 32, 6.188, 0.019 ],
[ 33, 6.187, 0.019 ],
[ 34, 6.194, 0.021 ],
[ 35, 6.192, 0.024 ],
[ 36, 6.193, 0.024 ],
[ 37, 6.187, 0.026 ],
[ 38, 6.184, 0.026 ],
[ 39, 6.183, 0.027 ],
[ 40, 6.189, 0.027 ] ] )
# Remove the rows whose first item is between 20 and 25
A = np.delete(A, np.where( np.bitwise_and( (A[:,0]>=20), (A[:,0]<=25) ) )[0], 0)
# Remove the rows whose first item is between 30 and 35
A = np.delete(A, np.where( np.bitwise_and( (A[:,0]>=30), (A[:,0]<=35) ) )[0], 0)
>>> A
array([[ 1.80000000e+01, 6.21500000e+00, 2.50000000e-02],
[ 1.90000000e+01, 6.20300000e+00, 2.50000000e-02],
[ 2.60000000e+01, 6.18900000e+00, 2.10000000e-02],
[ 2.70000000e+01, 6.18800000e+00, 2.00000000e-02],
[ 2.80000000e+01, 6.19200000e+00, 1.90000000e-02],
[ 2.90000000e+01, 6.18500000e+00, 2.00000000e-02],
[ 3.60000000e+01, 6.19300000e+00, 2.40000000e-02],
[ 3.70000000e+01, 6.18700000e+00, 2.60000000e-02],
[ 3.80000000e+01, 6.18400000e+00, 2.60000000e-02],
[ 3.90000000e+01, 6.18300000e+00, 2.70000000e-02],
[ 4.00000000e+01, 6.18900000e+00, 2.70000000e-02]])
The simplest way to delete rows and columns from arrays is the numpy.delete method.
Suppose I have the following array x:
x = array([[1,2,3],
[4,5,6],
[7,8,9]])
To delete the first row, do this:
x = numpy.delete(x, (0), axis=0)
To delete the third column, do this:
x = numpy.delete(x,(2), axis=1)
So you could find the indices of the rows which have a 0 in them, put them in a list or a tuple and pass this as the second argument of the function.
Here's a one liner (yes, it is similar to user333700's, but a little more straightforward):
>>> import numpy as np
>>> arr = np.array([[ 0.96488889, 0.73641667, 0.67521429, 0.592875, 0.53172222],
[ 0.78008333, 0.5938125, 0.481, 0.39883333, 0.]])
>>> print arr[arr.all(1)]
array([[ 0.96488889, 0.73641667, 0.67521429, 0.592875 , 0.53172222]])
By the way, this method is much, much faster than the masked array method for large matrices. For a 2048 x 5 matrix, this method is about 1000x faster.
By the way, user333700's method (from his comment) was slightly faster in my tests, though it boggles my mind why.
Numpy provides the where function:
import numpy as np
>>> x = np.array([1,2,3,4])
>>> x
array([1, 2, 3, 4])
>>> np.where(x <= 2)
(array([0, 1], dtype=int64),)
or
>>> x = np.arange(6).reshape(2, 3)
>>> x
array([[0, 1, 2],
[3, 4, 5]])
>>> x[np.where( x < 5 )]
array([0, 1, 2, 3, 4])
using where and delete in combination, you can for example delete the first row in the above matrix using this:
>>> np.delete(x, np.where(np.all(x < 3,axis=1)), axis=0)
array([[3, 4, 5]])
You can do this without numpy as well.
vec = [1, 2, 3, 4]
vec = [x for x in vec if x <=2]
vec
[1, 2]
Reference: https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions
you can change the indices_or_sections value to length of the first axis, this will prevent any empty arrays from being produced
import numpy as np
arr1 = np.array([[1.,2,3], [4,5,6], [7,8,9]])
arr_split = np.array_split(arr1,
indices_or_sections = arr1.shape[0],
axis = 0)
arr_split
>>> [
array([[1., 2., 3.]]),
array([[4., 5., 6.]]),
array([[7., 8., 9.]])
]
Just loop through and check the size. Only add them to the new list if they have a size greater than 0.
arr_split_new = [arr for arr in arr_split if arr.size > 0]
Note, numpy supports vectorized comparisons:
>>> test
array([[1, 2, 'a'],
[4, 5, 6],
[7, 'a', 9],
[10, 11, 12]], dtype=object)
>>> test == 'a'
array([[False, False, True],
[False, False, False],
[False, True, False],
[False, False, False]], dtype=bool)
Now, you want the rows where all are not equalt to 'a':
>>> (test != 'a').all(axis=1)
array([False, True, False, True], dtype=bool)
So, simply select the rows with the mask:
>>> row_mask = (test != 'a').all(axis=1)
>>> test[row_mask,:]
array([[4, 5, 6],
[10, 11, 12]], dtype=object)
Also, like this maybe? (Inspired from one of my another answers )
In [100]: mask = ~(test == 'a')
In [101]: mask
Out[101]:
array([[ True, True, False],
[ True, True, True],
[ True, False, True],
[ True, True, True]], dtype=bool)
In [102]: test[np.all(mask, axis=1), :]
Out[102]:
array([['4', '5', '6'],
['10', '11', '12']],
dtype='<U21')
But, please note that here we're not deleting any rows from the original array. We're just slicing out the rows which doesn't have the alphabet a.
You can easily do it with this piece of code:
new_data = data[(data == -1).sum(axis=1) < 2]
Result:
>>> new_data
array([[ 1.1, 1.2, 1.3, 1.4],
[ 2.1, 2.2, 2.3, -1. ]])
def remove_rows(data, threshold):
mask = np.array([np.sum(row == -1) < threshold for row in data])
return data[mask]
This function will return a new array with no rows having -1's more than or equal to the threshold
You need to pass in a Numpy array for it to work.
You can use something like this: first create dictionary of occurrences of each value in the sub arrays using np.unique and only keep arrays where no positive number appears more than once.
A = np.array([[-1, -1, -1, -1], [-1, -1, 1, 2], [-1, -1, 1, 1], [2, 1, -1, 2]])
new_array = []
# loop through each array
for array in A:
# Get a dictionary of the counts of each value
unique, counts = np.unique(array, return_counts=True)
counts = dict(zip(unique, counts))
# Find the number of occurences of postive numbers
positive_occurences = [value for key, value in counts.items() if key > 0]
# Append to new_array if no positive number appears more than once
if any(y > 1 for y in positive_occurences):
continue
else:
new_array.append(array)
new_array = np.array(new_array)
this returns:
array([[-1, -1, -1, -1],
[-1, -1, 1, 2]])
My fully-vectorized approach:
- sort each row
- detect duplicates by shifting the sorted array to the left by one and compare with itself
- mark rows with positive duplicates
- drop
import numpy as np
a = np.array([[-1, -1, -1, -1], [-1, -1, 1, 2], [-1, -1, 1, 1], [2, 1, -1, 2]])
# sort each row
b = np.sort(a)
# mark positive duplicates
drop = np.any((b[:,1:]>0) & (b[:,1:] == b[:,:-1]), axis=1)
# drop
aa = a[~drop, :]
Output:
array([[-1, -1, -1, -1],
[-1, -1, 1, 2]])
From your example, you want to remove row with index 1 from the first array,
and row with index 3 from the second array.
So use those indices when executing np.delete:
a1 = np.delete(i, z[0][0], axis=0)
np.argwhere will return both indices, but we're only interested in the rows:
np.argwhere(i > 8)[:, 0]
But really, we're only interested in unique rows, so we can take care of that too:
np.unique(np.argwhere(i > 8)[:, 0])
Altogether we get:
test = [np.array([[2,2,4],[10,3,5],[1,2,4,],[1,2,4]]),np.array([[1,2,3],[1,3,5],[6,3,1],[9,1,2]])]
for i in test:
z = np.unique(np.argwhere(i>8)[:, 0])
a1 = np.delete(i,z,axis=0)
print(a1)
#[[2 2 4]
# [1 2 4]
# [1 2 4]]
#[[1 2 3]
# [1 3 5]
# [6 3 1]]