append() creates a new array which can be the old array with the appended element.
I think it's more normal to use the proper method for adding an element:
a = numpy.append(a, a[0])
Answer from steabert on Stack Overflowappend() creates a new array which can be the old array with the appended element.
I think it's more normal to use the proper method for adding an element:
a = numpy.append(a, a[0])
When appending only once or once every now and again, using np.append on your array should be fine. The drawback of this approach is that memory is allocated for a completely new array every time it is called. When growing an array for a significant amount of samples it would be better to either pre-allocate the array (if the total size is known) or to append to a list and convert to an array afterward.
Using np.append:
b = np.array([0])
for k in range(int(10e4)):
b = np.append(b, k)
1.2 s ± 16.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Using python list converting to array afterward:
d = [0]
for k in range(int(10e4)):
d.append(k)
f = np.array(d)
13.5 ms ± 277 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Pre-allocating numpy array:
e = np.zeros((n,))
for k in range(n):
e[k] = k
9.92 ms ± 752 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
When the final size is unkown pre-allocating is difficult, I tried pre-allocating in chunks of 50 but it did not come close to using a list.
85.1 ms ± 561 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Issues with Optimization / Appending to a Numpy Array
python - How to append a NumPy array to a NumPy array - Stack Overflow
python - Optimal way to append to numpy array - Stack Overflow
python - How to append arrays to another numpy array? - Stack Overflow
Videos
https://paste.ofcode.org/zpT6HK22LuswbXhTHZfUK6
I am working on a starter project which involves storing pixel coordinates for various "territories" (dictionaries) on a map. Because there are a huge amount of coordinates (33 million pixels), I have been trying to optimize the program so it uses less memory. I have been using numpy's int16 type and ndarray to save memory. The problem I am getting however, is with adding pixels to territories. Each territory (dictionary) has a "pixels" key with an array as a value, and to add pixels to that array I use the following method:
def addPixel(territory, pixel):
pixels = getProperty(territory, "pixels")
territory["pixels"] = numpy.append(pixels, pixel)This method, however, appears to be very slow, and it would take a huge amount of time to iterate over millions of pixels. Does anyone know of a faster way the append items to ndarrays?
Nest the arrays so that they have more than one axis, and then specify the axis when using append.
import numpy as np
a = np.array([[1, 2]]) # note the braces
b = np.array([[3, 4]])
c = np.array([[5, 6]])
d = np.append(a, b, axis=0)
print(d)
# [[1 2]
# [3 4]]
e = np.append(d, c, axis=0)
print(e)
# [[1 2]
# [3 4]
# [5 6]]
Alternately, if you stick with lists, use numpy.vstack:
import numpy as np
a = [1, 2]
b = [3, 4]
c = [5, 6]
d = np.vstack([a, b])
print(d)
# [[1 2]
# [3 4]]
e = np.vstack([d, c])
print(e)
# [[1 2]
# [3 4]
# [5 6]]
I found it handy to use this code with numpy. For example:
loss = None
new_coming_loss = [0, 1, 0, 0, 1]
loss = np.concatenate((loss, [new_coming_loss]), axis=0) if loss is not None else [new_coming_loss]
Practical Use:
self.epoch_losses = None
self.epoch_losses = np.concatenate((self.epoch_losses, [loss.flatten()]), axis=0) if self.epoch_losses is not None else [loss.flatten()]
Copy and paste solution:
def append(list, element):
return np.concatenate((list, [element]), axis=0) if list is not None else [element]
WARNING: the dimension of list and element should be the same except the first dimension, otherwise you will get:
ValueError: all the input array dimensions except for the concatenation axis must match exactly
Appending to numpy arrays is very inefficient. This is because the interpreter needs to find and assign memory for the entire array at every single step. Depending on the application, there are much better strategies.
If you know the length in advance, it is best to pre-allocate the array using a function like np.ones, np.zeros, or np.empty.
desired_length = 500
results = np.empty(desired_length)
for i in range(desired_length):
results[i] = i**2
If you don't know the length, it's probably more efficient to keep your results in a regular list and convert it to an array afterwards.
results = []
while condition:
a = do_stuff()
results.append(a)
results = np.array(results)
Here are some timings on my computer.
def pre_allocate():
results = np.empty(5000)
for i in range(5000):
results[i] = i**2
return results
def list_append():
results = []
for i in range(5000):
results.append(i**2)
return np.array(results)
def numpy_append():
results = np.array([])
for i in range(5000):
np.append(results, i**2)
return results
%timeit pre_allocate()
# 100 loops, best of 3: 2.42 ms per loop
%timeit list_append()
# 100 loops, best of 3: 2.5 ms per loop
%timeit numpy_append()
# 10 loops, best of 3: 48.4 ms per loop
So you can see that both pre-allocating and using a list then converting are much faster.
If you know the size of the array at the end of the run, then it is going to be much faster to pre-allocate an array of the appropriate size and then set the values. If you do need to append on-the fly, it's probably better to try to not do this one element at a time, instead appending as few times as possible to avoid generating many copies over and over again. You might also want to do some profiling of the difference in timings of np.append, np.hstack, np.concatenate. etc.
The problem that stack functions don't work, is that they need that the row added is of the same size of the already present rows. Using np.array([[]]), the first row is has a length of zero, which means that you can only add rows that also have length zero.
In order to solve this, we need to tell Numpy that the first row is of size two and not zero. The array thus needs to be of size (0, 2) and not (0, 0). This can be done using one of the array-initializing functions that accept size arguments, like empty, zeros or ones. Which function does not matter, as there are no spaces to fill.
Then you can use one of the functions mentioned in comments, like vstack or stack. The code thus becomes:
import numpy as np
all_coordinates = np.zeros((0, 2))
for y in range(2):
for x in range(2):
coordinate = np.array([[x,y]])
# append
all_coordinates = np.vstack((all_coordinates, coordinate))
print(all_coordinates)
In such a case, I would use a list and only convert it into an array once you have appended all the elements you want. here is a suggested improvement
import numpy as np
all_coordinates = []
for y in range(2):
for x in range(2):
coordinate = np.array([x,y])
# append
all_coordinates.append(coordinate)
all_coordinates = np.array(all_coordinates)
print(all_coordinates)
The output of this code is indeed
array([[0, 0],
[1, 0],
[0, 1],
[1, 1]])