Appending to numpy arrays is very inefficient. This is because the interpreter needs to find and assign memory for the entire array at every single step. Depending on the application, there are much better strategies.
If you know the length in advance, it is best to pre-allocate the array using a function like np.ones, np.zeros, or np.empty.
desired_length = 500
results = np.empty(desired_length)
for i in range(desired_length):
results[i] = i**2
If you don't know the length, it's probably more efficient to keep your results in a regular list and convert it to an array afterwards.
results = []
while condition:
a = do_stuff()
results.append(a)
results = np.array(results)
Here are some timings on my computer.
def pre_allocate():
results = np.empty(5000)
for i in range(5000):
results[i] = i**2
return results
def list_append():
results = []
for i in range(5000):
results.append(i**2)
return np.array(results)
def numpy_append():
results = np.array([])
for i in range(5000):
np.append(results, i**2)
return results
%timeit pre_allocate()
# 100 loops, best of 3: 2.42 ms per loop
%timeit list_append()
# 100 loops, best of 3: 2.5 ms per loop
%timeit numpy_append()
# 10 loops, best of 3: 48.4 ms per loop
So you can see that both pre-allocating and using a list then converting are much faster.
Answer from Roger Fan on Stack Overflowpython - Optimal way to append to numpy array - Stack Overflow
python - Concatenate a NumPy array to another NumPy array - Stack Overflow
Issues with Optimization / Appending to a Numpy Array
Numpy Array Append
Videos
Appending to numpy arrays is very inefficient. This is because the interpreter needs to find and assign memory for the entire array at every single step. Depending on the application, there are much better strategies.
If you know the length in advance, it is best to pre-allocate the array using a function like np.ones, np.zeros, or np.empty.
desired_length = 500
results = np.empty(desired_length)
for i in range(desired_length):
results[i] = i**2
If you don't know the length, it's probably more efficient to keep your results in a regular list and convert it to an array afterwards.
results = []
while condition:
a = do_stuff()
results.append(a)
results = np.array(results)
Here are some timings on my computer.
def pre_allocate():
results = np.empty(5000)
for i in range(5000):
results[i] = i**2
return results
def list_append():
results = []
for i in range(5000):
results.append(i**2)
return np.array(results)
def numpy_append():
results = np.array([])
for i in range(5000):
np.append(results, i**2)
return results
%timeit pre_allocate()
# 100 loops, best of 3: 2.42 ms per loop
%timeit list_append()
# 100 loops, best of 3: 2.5 ms per loop
%timeit numpy_append()
# 10 loops, best of 3: 48.4 ms per loop
So you can see that both pre-allocating and using a list then converting are much faster.
If you know the size of the array at the end of the run, then it is going to be much faster to pre-allocate an array of the appropriate size and then set the values. If you do need to append on-the fly, it's probably better to try to not do this one element at a time, instead appending as few times as possible to avoid generating many copies over and over again. You might also want to do some profiling of the difference in timings of np.append, np.hstack, np.concatenate. etc.
In [1]: import numpy as np
In [2]: a = np.array([[1, 2, 3], [4, 5, 6]])
In [3]: b = np.array([[9, 8, 7], [6, 5, 4]])
In [4]: np.concatenate((a, b))
Out[4]:
array([[1, 2, 3],
[4, 5, 6],
[9, 8, 7],
[6, 5, 4]])
or this:
In [1]: a = np.array([1, 2, 3])
In [2]: b = np.array([4, 5, 6])
In [3]: np.vstack((a, b))
Out[3]:
array([[1, 2, 3],
[4, 5, 6]])
Well, the error message says it all: NumPy arrays do not have an append() method. There's a free function numpy.append() however:
numpy.append(M, a)
This will create a new array instead of mutating M in place. Note that using numpy.append() involves copying both arrays. You will get better performing code if you use fixed-sized NumPy arrays.
https://paste.ofcode.org/zpT6HK22LuswbXhTHZfUK6
I am working on a starter project which involves storing pixel coordinates for various "territories" (dictionaries) on a map. Because there are a huge amount of coordinates (33 million pixels), I have been trying to optimize the program so it uses less memory. I have been using numpy's int16 type and ndarray to save memory. The problem I am getting however, is with adding pixels to territories. Each territory (dictionary) has a "pixels" key with an array as a value, and to add pixels to that array I use the following method:
def addPixel(territory, pixel):
pixels = getProperty(territory, "pixels")
territory["pixels"] = numpy.append(pixels, pixel)This method, however, appears to be very slow, and it would take a huge amount of time to iterate over millions of pixels. Does anyone know of a faster way the append items to ndarrays?
I'm trying to find out how to append a list to a Numpy array.
So I have a list of lists that I then convert into a Numpy array. I then want to append a list to that array.
So let's say I have
list = [[a,b,c],[b,c,d],[a,d,b]]
I first do:
list = np.array(list)
... (code that does calculations on that numpy array)
I then want to append:
list2 = [c,b,a]
to list.
I've tried np.index as well as np.append, however both of them give me the following output:
list = list = [[a,b,c],[b,c,d],[a,d,b,c,b,a]]
when I want:
list = [[a,b,c],[b,c,d],[a,d,b],[c,b,a]]
If this is too convoluted let me know, I can try explain it better.
I appreciate the help.
» pip install npy-append-array