Brave Search

stackoverflow.com › questions › 37593013 › deep-copy-of-a-np-array-of-np-array

import numpy as np
import copy

pairs = [(2, 3), (3, 4), (4, 5)]
array_of_arrays = np.array([np.arange(a*b).reshape(a,b) for (a, b) in pairs])

a = copy.deepcopy(array_of_arrays)

Feel free to read up more about this here.

Oh, here is simplest test case:

a[0][0,0]
print a[0][0,0], array_of_arrays[0][0,0]

Answer from Tomasz Plaskota on Stack Overflow

Stack Overflow

stackoverflow.com › questions › 37593013 › deep-copy-of-a-np-array-of-np-array

python - Deep copy of a np.array of np.array - Stack Overflow

Top answer

1 of 7

34

import numpy as np
import copy

pairs = [(2, 3), (3, 4), (4, 5)]
array_of_arrays = np.array([np.arange(a*b).reshape(a,b) for (a, b) in pairs])

a = copy.deepcopy(array_of_arrays)

Feel free to read up more about this here.

Oh, here is simplest test case:

a[0][0,0]
print a[0][0,0], array_of_arrays[0][0,0]

2 of 7

5

In [276]: array_of_arrays
Out[276]: 
array([array([[0, 1, 2],
       [3, 4, 5]]),
       array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]]),
       array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])], dtype=object)

array_of_arrays is dtype=object; that means each element of the array is a pointer to an object else where in memory. In this case those elements are arrays of different sizes.

a = array_of_arrays[:]

a is a new array, but a view of array_of_arrays; that is, it has the same data buffer (which in this case is list of pointers).

b = array_of_arrays[:][:]

this is just a view of a view. The second [:] acts on the result of the first.

c = np.array(array_of_arrays, copy=True)

This is the same as array_of_arrays.copy(). c has a new data buffer, a copy of the originals

If I replace an element of c, it will not affect array_of_arrays:

c[0] = np.arange(3)

But if I modify an element of c, it will modify the same element in array_of_arrays - because they both point to the same array.

The same sort of thing applies to nested lists of lists. What array adds is the view case.

d = np.array([np.array(x, copy=True) for x in array_of_arrays])

In this case you are making copies of the individual elements. As others noted there is a deepcopy function. It was designed for things like lists of lists, but works on arrays as well. It is basically doing what you do with d; recursively working down the nesting tree.

In general, an object array is like list nesting. A few operations cross the object boundary, e.g.

 array_of_arrays+1

but even this effectively is

np.array([x+1 for x in array_of_arrays])

One thing that a object array adds, compared to a list, is operations like reshape. array_of_arrays.reshape(3,1) makes it 2d; if it had 4 elements you could do array_of_arrays.reshape(2,2). Some times that's handy; other times it's a pain (it's harder to iterate).

numpy - explanation of "np.copy is a shallow copy and will not copy object elements within arrays"

python - Numpy Deep Copy Array

arrays - Equality of copy.copy and copy.deepcopy in python copy module

python - How can I manipulate a copy of an array without changing the original array?

Python copy.copy() vs NumPy np.copy()

More results from stackoverflow.com

reddit.com › r/learnpython › do i need to be using deepcopy with numpy arrays in this situation?

r/learnpython on Reddit: Do I need to be using deepcopy with numpy arrays in this situation?

April 21, 2023 -

I've done a ton of reading about this and I sort of know the difference between the two, but I still don't know when I need to use copy.copy() vs copy.deepcopy(). It seems like copy.copy() makes a new object of the "container" (for instance, if it is a list then you get a new list object) but then populates this container with the references to the original element objects (so each element in the two different lists point to the same spot in memory).

Essentially, my question is that I have a bunch of numpy arrays and I don't think I'm using copy correctly (or if I need ot be using it at all). It seems like if I am only doing assignments or using these values to set other values (eg output = self.weights*input or whatever) then I don't need to make copies (its fine that the references point to the same spot in memory since I just need read not write access), but if I am doing things like increments / decrements or setting the new value based on the previous value, then I would be changing the values in memory and these are shared by all my objects? Some code:

For instance, let's say I have some Parent class and it has its children Child objects and the parent sends its models to the children and the children update the models based on their own data. Specifically, each child should get (a copy?) the parent's weight matrix (numpy array) to use as initialization, and then start updating its own matrix (not shared with any other children or the parent).

parent = Parent()
child1 = Child()
child2 = Child()
...
childN = Child()

for child in children_list:
    child.weights = parent.weights  # copy? deepcopy?

for child in children_list:
    for _ in num_grad_steps:
        child.weights -= child.learning_rate * child.gradient(child.weights, child.input_data)
    
    # Send the updated weights back to the parent
    parent.new_weights_list.append(child.weights)

In this case, child.weights is being updated by each child (note I don't want a single matrix that is updated by all children, I want one matrix as initialization, then every child runs with it so I end up with N unique final matrices), and if it is either just the same object (basic assignment: child.weights = self.weights) or just a shallow copy (child.weights = copy.copy(parent.weights)) then FOR ALL THE CHILDREN the weights are shared instead of each child getting the same initial matrix and privately updating its own copy, right? If I instead switch this to child.weights = copy.deepcopy(parent.weights) then I think this fixes it, but since my weights matrix is pretty large this just takes an extremely long time to run (and it seems like I'm missing something, like it shouldn't be this hard. I feel like code I see online in similar applications doesn't do this, but many don't have the weights distributed across so many objects at once I guess?). Do I need to be using deepcopy or is there something I'm missing? Greatly appreciate any help!

Top answer

1 of 1

2

The short answer is that if you have, say, a numpy array of ints, floats, etc (which is usually) and you want to make a separate copy where the two will not interact in any way (which is what it sounds like), there's no need to deepcopy, and you can just use the array.copy() method. Long version: You are also correct on copy vs deepcopy. If you use copy on, say, a list (more on arrays later), you will get a different list containing the same elements of the original. So: l = [[1], [2]] l2 = copy.copy(l) # l2 is different from l # The sublists inside l2 are the same as the ones in l l2[0] = 42 # l unchanged, you modified l2 which is different l2[1][0] = 1337 # reflected in l, because l2[1] is the same as l[1] So in this case, if you didn't want l2[1][0] = 1337 to be reflected in l, you'd use deepcopy. This would ensure that the sublists are different, as well as the outer list. However, if the contents of l are say, floats or ints, then this doesn't matter. Floats are immutable - you can't change em. What you might think of as modifying a float is actually creating a new float and putting that new float somewhere. So: l = [1.2, 3.4] l2 = l.copy() # same as copy.copy(l) # The elements of l2 are still the same as of l as before. But: l2[0] += 7 # does not actually modify the # float 1.2, but creates a new # float 8.2 and assigns that to # l2[0] - which modifies l2, which # is different from l, so l is unchanged So in this case deepcopy is not necessary to make l and l2 behave completely distinctly (though it doesn't hurt anything - might use a bit more ram depending on what types are in l). Numpy arrays are almost certainly a bit different though. For one thing, a single numpy array with shape (5, 5), say, is not actually an array of arrays, but a single sequence of length 25 that uses math to let you use double indexing. Also, under the hood numpy arrays have byte buffers representing whatever types, so I strongly suspect that array.copy just copies that buffer (at least if it's an array of numbers). So it's possible that it's not really the same rules as lists of objects anyway. But .copy is certainly sufficient for what you want.

Discussions

Why is copying a list so damn difficult in python?

This kind of confusion is mostly because you're thinking about python's object model wrong, or don't fully understand it. Perhaps this is a better way to think about things. Say you have this data structure: x = [ [1,2,3], [4,5,6], ] This statement creates 3 lists: 2 inner lists and one outer list. A reference to the outer list is then made available under the name x. When you execute this statement: y = x no data gets copied. You still have the same 3 lists in memory somewhere. All this did is make the outer list availible under the name y, in addition to its previous name x. When you execute this statement: y = list(x) or y = x[:] This creates a new list with the same contents as x. List x contained a reference to the 2 inner lists, so the new list will also contain a reference to those same 2 inner lists. Only one list is copied—the outer list. Now there are 4 lists in memory, the two inner lists, the outer list, and the copy of the outer list. The original outer list is available under the name x, and the new outer list is made available under the name y. The inner lists have not been copied! You can access and edit the inner lists from either x or y at this point! If you have a two dimensional (or higher) list, or any kind of nested data structure, and you want to make a full copy of everything, then you want to use the deepcopy() function in the copy module. Your solution also works for 2-D lists, as iterates over the items in the outer list and makes a copy of each of them, then builds a new outer list for all the inner copies. Hope this explanation helps. More on reddit.com

r/learnpython

30

9

March 16, 2013

Do I need to be using deepcopy with numpy arrays in this situation?

The short answer is that if you have, say, a numpy array of ints, floats, etc (which is usually) and you want to make a separate copy where the two will not interact in any way (which is what it sounds like), there's no need to deepcopy, and you can just use the array.copy() method. Long version: You are also correct on copy vs deepcopy. If you use copy on, say, a list (more on arrays later), you will get a different list containing the same elements of the original. So: l = [[1], [2]] l2 = copy.copy(l) # l2 is different from l # The sublists inside l2 are the same as the ones in l l2[0] = 42 # l unchanged, you modified l2 which is different l2[1][0] = 1337 # reflected in l, because l2[1] is the same as l[1] So in this case, if you didn't want l2[1][0] = 1337 to be reflected in l, you'd use deepcopy. This would ensure that the sublists are different, as well as the outer list. However, if the contents of l are say, floats or ints, then this doesn't matter. Floats are immutable - you can't change em. What you might think of as modifying a float is actually creating a new float and putting that new float somewhere. So: l = [1.2, 3.4] l2 = l.copy() # same as copy.copy(l) # The elements of l2 are still the same as of l as before. But: l2[0] += 7 # does not actually modify the # float 1.2, but creates a new # float 8.2 and assigns that to # l2[0] - which modifies l2, which # is different from l, so l is unchanged So in this case deepcopy is not necessary to make l and l2 behave completely distinctly (though it doesn't hurt anything - might use a bit more ram depending on what types are in l). Numpy arrays are almost certainly a bit different though. For one thing, a single numpy array with shape (5, 5), say, is not actually an array of arrays, but a single sequence of length 25 that uses math to let you use double indexing. Also, under the hood numpy arrays have byte buffers representing whatever types, so I strongly suspect that array.copy just copies that buffer (at least if it's an array of numbers). So it's possible that it's not really the same rules as lists of objects anyway. But .copy is certainly sufficient for what you want. More on reddit.com

r/learnpython

1

April 21, 2023

Error Help: ValueError: Object Too Deep for Desired Array

Posted by u/Mathematical-Balloon - 2 votes and 1 comment More on reddit.com

r/learnprogramming

1

2

July 20, 2022

Deep learning: Memory error with arrays and lists in python

First of all, 25396 images seems like a very low number, especially if you're building your own CNN. The n/w will most likely always overfit to your data and pick up sampling errors. Karpathy himself has quoted - "Don't try to be a hero". Most of the time, you'll be battling the variance, which will just result in you dumbing down your own network. Have you tried transfer learning?

Regarding the memory problem - It's pretty obvious. You're loading two tensors of shapes (25369, 204, 204, 3) and (25369, 39) directly into memory, which seems to be too much. If you're using Keras, use the ImageDataGenerator utility and it's flow_from_directory method. You can stream images from folders with a specific batch size - very handy while handling a large number of images.

More on reddit.com

r/deeplearning

4

5

May 29, 2018

Videos

youtube.com

Copy vs Views Numpy Python Array || NumPy Copy vs View ...