There are a few alternatives. The below assumes you are iterating over a 1d NumPy array.
Iterate with range
for j in range(theta.shape[0]): # or range(len(theta))
some_function(j, theta[j], theta)
Note this is the only of the 3 solutions which will work with numba. This is noteworthy since iterating over a NumPy array explicitly is usually only efficient when combined with numba or another means of pre-compilation.
Iterate with enumerate
for idx, j in enumerate(theta):
some_function(idx, j, theta)
The most efficient of the 3 solutions for 1d arrays. See benchmarking below.
Iterate with np.ndenumerate
for idx, j in np.ndenumerate(theta):
some_function(idx[0], j, theta)
Notice the additional indexing step in idx[0]. This is necessary since the index (like shape) of a 1d NumPy array is given as a singleton tuple. For a 1d array, np.ndenumerate is inefficient; its benefits only show for multi-dimensional arrays.
Performance benchmarking
# Python 3.7, NumPy 1.14.3
np.random.seed(0)
arr = np.random.random(10**6)
def enumerater(arr):
for index, value in enumerate(arr):
index, value
pass
def ranger(arr):
for index in range(len(arr)):
index, arr[index]
pass
def ndenumerater(arr):
for index, value in np.ndenumerate(arr):
index[0], value
pass
%timeit enumerater(arr) # 131 ms
%timeit ranger(arr) # 171 ms
%timeit ndenumerater(arr) # 579 ms
There are a few alternatives. The below assumes you are iterating over a 1d NumPy array.
Iterate with range
for j in range(theta.shape[0]): # or range(len(theta))
some_function(j, theta[j], theta)
Note this is the only of the 3 solutions which will work with numba. This is noteworthy since iterating over a NumPy array explicitly is usually only efficient when combined with numba or another means of pre-compilation.
Iterate with enumerate
for idx, j in enumerate(theta):
some_function(idx, j, theta)
The most efficient of the 3 solutions for 1d arrays. See benchmarking below.
Iterate with np.ndenumerate
for idx, j in np.ndenumerate(theta):
some_function(idx[0], j, theta)
Notice the additional indexing step in idx[0]. This is necessary since the index (like shape) of a 1d NumPy array is given as a singleton tuple. For a 1d array, np.ndenumerate is inefficient; its benefits only show for multi-dimensional arrays.
Performance benchmarking
# Python 3.7, NumPy 1.14.3
np.random.seed(0)
arr = np.random.random(10**6)
def enumerater(arr):
for index, value in enumerate(arr):
index, value
pass
def ranger(arr):
for index in range(len(arr)):
index, arr[index]
pass
def ndenumerater(arr):
for index, value in np.ndenumerate(arr):
index[0], value
pass
%timeit enumerater(arr) # 131 ms
%timeit ranger(arr) # 171 ms
%timeit ndenumerater(arr) # 579 ms
You can use numpy.ndenumerate for example
import numpy as np
test_array = np.arange(2, 3, 0.1)
for index, value in np.ndenumerate(test_array):
print(index[0], value)
For more information refer to https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndenumerate.html
As I've become more familiar with the NumPy and general scientific array processing ecosystem, it's become clearer to me that iteration is usually outright wrong due to how slow it is in comparison, and writing to use a vectorized operation is best whenever possible. Though the style is not as obvious/Pythonic at first, I've (anecdotally) gained ridiculous speedups with vectorized operations; more than 1000x in a case of swapping out a form like some row iteration .apply(lambda)
@MSeifert's answer much better provides this and will be significantly more performant on a dataset of any real size
More general Answer by @cs95 covering and comparing alternatives to iteration in Pandas
Original Answer
You can iterate through the values in your array with numpy.ndenumerate to get the indices of the values in your array.
Using the documentation above:
A = np.array([[1,2,3],[4,5,6],[7,8,9]])
for index, values in np.ndenumerate(A):
print(index, values) # operate here
You can do it using np.ndenumerate but generally you don't need to iterate over an array.
You can simply create a meshgrid (or open grid) to get all indices at once and you can then process them (vectorized) much faster.
For example
>>> x, y = np.mgrid[slice(A.shape[0]), slice(A.shape[1])]
>>> x
array([[0, 0, 0],
[1, 1, 1],
[2, 2, 2]])
>>> y
array([[0, 1, 2],
[0, 1, 2],
[0, 1, 2]])
and these can be processed like any other array. So if your function that needs the indices can be vectorized you shouldn't do the manual loop!
For example to calculate the lattice distance for each point to a point say (2, 3):
>>> abs(x - 2) + abs(y - 3)
array([[5, 4, 3],
[4, 3, 2],
[3, 2, 1]])
For distances an ogrid would be faster. Just replace np.mgrid with np.ogrid:
>>> x, y = np.ogrid[slice(A.shape[0]), slice(A.shape[1])]
>>> np.hypot(x - 2, y - 3) # cartesian distance this time! :-)
array([[ 3.60555128, 2.82842712, 2.23606798],
[ 3.16227766, 2.23606798, 1.41421356],
[ 3. , 2. , 1. ]])
To take full advantage of numpy's speed, you want to create ufuncs whenever possible. Applying vectorize to a function as mgibsonbr suggests is one way to do that, but a better way, if possible, is simply to construct a function that takes advantage of numpy's built-in ufuncs. So something like this:
>>> import numpy
>>> a = numpy.random.random(10)
>>> a + 1
array([ 1.29738145, 1.33004628, 1.45825441, 1.46171177, 1.56863326,
1.58502855, 1.06693054, 1.93304272, 1.66056379, 1.91418473])
>>> (a + 1) * 0.25 / 4
array([ 0.08108634, 0.08312789, 0.0911409 , 0.09135699, 0.09803958,
0.09906428, 0.06668316, 0.12081517, 0.10378524, 0.11963655])
What is the nature of the function you want to apply across the numpy array? If you tell us, perhaps we can help you come up with a version that uses only numpy ufuncs.
It's also possible to generate an array of indices without using enumerate. Numpy provides ndenumerate, which is an iterator, and probably slower, but it also provides indices, which is a very quick way to generate the indices corresponding to the values in an array. So...
>>> numpy.indices(a.shape)
array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])
So to be more explicit, you can use the above and combine them using numpy.rec.fromarrays:
>>> a = numpy.random.random(10)
>>> ind = numpy.indices(a.shape)
>>> numpy.rec.fromarrays([ind[0], a])
rec.array([(0, 0.092473494150913438), (1, 0.20853257641948986),
(2, 0.35141455604686067), (3, 0.12212258656960817),
(4, 0.50986868372639049), (5, 0.0011439325711705139),
(6, 0.50412473457942508), (7, 0.28973489788728601),
(8, 0.20078799423168536), (9, 0.34527678271856999)],
dtype=[('f0', '<i8'), ('f1', '<f8')])
It's starting to sound like your main concern is performing the operation in-place. That's harder to do using vectorize but it's easy with the ufunc approach:
>>> def somefunc(a):
... a += 1
... a /= 15
...
>>> a = numpy.random.random(10)
>>> b = a
>>> somefunc(a)
>>> a
array([ 0.07158446, 0.07052393, 0.07276768, 0.09813235, 0.09429439,
0.08561703, 0.11204622, 0.10773558, 0.11878885, 0.10969279])
>>> b
array([ 0.07158446, 0.07052393, 0.07276768, 0.09813235, 0.09429439,
0.08561703, 0.11204622, 0.10773558, 0.11878885, 0.10969279])
As you can see, numpy performs these operations in-place.
Check numpy.vectorize, it should let you apply arbitrary functions to numpy arrays. For your simple example, you'd do something like this:
vecFunc = vectorize(lambda x: x + 1)
vecFunc(y)
However, that will create a new numpy array instead of modifying it in-place (which may or may not be a problem in your particular case).
In general, you'll always be better manipulating numpy structures with numpy functions than iterating with python functions, since the former are not only optimized but implemented in C, while the latter will be always interpreted.