Numpy arrays are poorly suited to appending because they’re fixed-length, and doing an append operation copies the entire array to a new array with one more spot, and then puts the new thing in that spot. If your main operation is appending then you’re likely better off with standard Python lists, or alternatively you can implement your own appending algorithm that does larger resizes (e.g. double each time) and separately keep track of how many elements have been put into each territory so that you only need to resize when you run out of space again. More generally, it’s not clear what you’re actually doing here. It sounds like you’re using a sparse array type approach where you only store individual position coordinates, but if you’re in charge of managing the whole map maybe you’re better off just having a single map array that keeps track of which territory ‘owns’ each pixel? Note that numpy is optimised for vectorised operations that apply to a large portion of an array at the same time. If you find yourself looping through a numpy array there’s probably something that could be handled better with vectorised operations, and if not it’s possible a numpy array is the wrong data structure to use :-) More on reddit.com

Just so you know: A numpy array is not made to be appended to like a list is. In the background of np.append an entire new array has to be created, so it's expensive (slow). It's much better to use lists if you need to append data. More on reddit.com

Vectorize your calculations so they apply simultaneously to all rows and then use np.concatenate to create a combined array: new_columns = # Calculate new columns by only using array operations on 'img' to create a (120560400, 6) array img = np.concatenate([img, new_columns], axis=1) # (120560400, 12) array For more specific advice, I'd need to know how you actually calculate the new columns. More on reddit.com

I've done exactly this before. It depends on if you want to read them in a different order than you are going to write them. If you're okay with reading in the same order you write, the solution is easy:

out_file = open('filename', 'ab')
np.save(out_file, some_ndarray)

You may want to open then close the file before in 'w' mode to make sure you're not appending on a huge mess. To read the arrays, just open the file and repeatedly call np.load on the file object.

More on reddit.com