The way to "start" the array that you want is:
arr = np.empty((0,3), int)
Which is an empty array but it has the proper dimensionality.
>>> arr
array([], shape=(0, 3), dtype=int64)
Then be sure to append along axis 0:
arr = np.append(arr, np.array([[1,2,3]]), axis=0)
arr = np.append(arr, np.array([[4,5,6]]), axis=0)
But, @jonrsharpe is right. In fact, if you're going to be appending in a loop, it would be much faster to append to a list as in your first example, then convert to a numpy array at the end, since you're really not using numpy as intended during the loop:
In [210]: %%timeit
.....: l = []
.....: for i in xrange(1000):
.....: l.append([3*i+1,3*i+2,3*i+3])
.....: l = np.asarray(l)
.....:
1000 loops, best of 3: 1.18 ms per loop
In [211]: %%timeit
.....: a = np.empty((0,3), int)
.....: for i in xrange(1000):
.....: a = np.append(a, 3*i+np.array([[1,2,3]]), 0)
.....:
100 loops, best of 3: 18.5 ms per loop
In [214]: np.allclose(a, l)
Out[214]: True
The numpythonic way to do it depends on your application, but it would be more like:
In [220]: timeit n = np.arange(1,3001).reshape(1000,3)
100000 loops, best of 3: 5.93 µs per loop
In [221]: np.allclose(a, n)
Out[221]: True
The way to "start" the array that you want is:
arr = np.empty((0,3), int)
Which is an empty array but it has the proper dimensionality.
>>> arr
array([], shape=(0, 3), dtype=int64)
Then be sure to append along axis 0:
arr = np.append(arr, np.array([[1,2,3]]), axis=0)
arr = np.append(arr, np.array([[4,5,6]]), axis=0)
But, @jonrsharpe is right. In fact, if you're going to be appending in a loop, it would be much faster to append to a list as in your first example, then convert to a numpy array at the end, since you're really not using numpy as intended during the loop:
In [210]: %%timeit
.....: l = []
.....: for i in xrange(1000):
.....: l.append([3*i+1,3*i+2,3*i+3])
.....: l = np.asarray(l)
.....:
1000 loops, best of 3: 1.18 ms per loop
In [211]: %%timeit
.....: a = np.empty((0,3), int)
.....: for i in xrange(1000):
.....: a = np.append(a, 3*i+np.array([[1,2,3]]), 0)
.....:
100 loops, best of 3: 18.5 ms per loop
In [214]: np.allclose(a, l)
Out[214]: True
The numpythonic way to do it depends on your application, but it would be more like:
In [220]: timeit n = np.arange(1,3001).reshape(1000,3)
100000 loops, best of 3: 5.93 µs per loop
In [221]: np.allclose(a, n)
Out[221]: True
Here is my solution:
arr = []
arr.append([1,2,3])
arr.append([4,5,6])
np_arr = np.array(arr)
That is the wrong mental model for using NumPy efficiently. NumPy arrays are stored in contiguous blocks of memory. To append rows or columns to an existing array, the entire array needs to be copied to a new block of memory, creating gaps for the new elements to be stored. This is very inefficient if done repeatedly.
Instead of appending rows, allocate a suitably sized array, and then assign to it row-by-row:
>>> import numpy as np
>>> a = np.zeros(shape=(3, 2))
>>> a
array([[ 0., 0.],
[ 0., 0.],
[ 0., 0.]])
>>> a[0] = [1, 2]
>>> a[1] = [3, 4]
>>> a[2] = [5, 6]
>>> a
array([[ 1., 2.],
[ 3., 4.],
[ 5., 6.]])
A NumPy array is a very different data structure from a list and is designed to be used in different ways. Your use of hstack is potentially very inefficient... every time you call it, all the data in the existing array is copied into a new one. (The append function will have the same issue.) If you want to build up your matrix one column at a time, you might be best off to keep it in a list until it is finished, and only then convert it into an array.
e.g.
mylist = []
for item in data:
mylist.append(item)
mat = numpy.array(mylist)
item can be a list, an array or any iterable, as long
as each item has the same number of elements.
In this particular case (data is some iterable holding the matrix columns) you can simply use
mat = numpy.array(data)
(Also note that using list as a variable name is probably not good practice since it masks the built-in type by that name, which can lead to bugs.)
EDIT:
If for some reason you really do want to create an empty array, you can just use numpy.array([]), but this is rarely useful!
1. A Hack
You can create an empty array using the np.empty() function and specify the dimensions to be (0, 0) and then use np.append() to append items later.
>>> a = np.empty((0, 0))
>>> a
array([], shape=(0, 0), dtype=float64)
>>> b = np.append(a, [1, 2])
>>> b
array([1., 2.])
2. However...
The hack above is not advisable, use it with caution. Appending to lists has O(N) complexity, while appending to arrays has O(N^2) (besides different memory use). The proper way should be, then, to append to lists. Note that using list() on numpy arrays to transform them into lists is not correct, as you will get a list of numpy arrays. Instead, use the .tolist() method.
>>> a = np.array([[1, 2], [3, 4]])
>>>
>>> list(a)
[array([1, 2]), array([3, 4])]
>>>
>>> a.tolist()
[[1, 2], [3, 4]]
It is advised to use normal python lists and then at the end convert the list to a numpy array.
x = np.asarray(x)
I suspect you are trying to replicate this working list code:
In [56]: x = []
In [57]: x.append([1,2])
In [58]: x
Out[58]: [[1, 2]]
In [59]: np.array(x)
Out[59]: array([[1, 2]])
But with arrays:
In [53]: x = np.empty((2,2),int)
In [54]: x
Out[54]:
array([[73096208, 10273248],
[ 2, -1]])
Despite the name, the np.empty array is NOT a close of the empty list. It has 4 elements, the shape that you specified.
In [55]: np.append(x, np.array([1,2]), axis=0)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-55-64dd8e7900e3> in <module>
----> 1 np.append(x, np.array([1,2]), axis=0)
<__array_function__ internals> in append(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/numpy/lib/function_base.py in append(arr, values, axis)
4691 values = ravel(values)
4692 axis = arr.ndim-1
-> 4693 return concatenate((arr, values), axis=axis)
4694
4695
<__array_function__ internals> in concatenate(*args, **kwargs)
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 1 has 1 dimension(s)
Note that np.append has passed the task on to np.concatenate. With the axis parameter, that's all this append does. It is NOT a list append clone.
np.concatenate demands consistency in the dimensions of its inputs. One is (2,2), the other (2,). Mismatched dimensions.
np.append is a dangerous function, and not that useful even when used correctly. np.concatenate (and the various stack) functions are useful. But you need to pay attention to shapes. And don't use them iteratively. List append is more efficient for that.
When you got this error, did you look up the np.append, np.empty (and np.concatenate) functions? Read and understand the docs? In the long run SO questions aren't a substitute for reading the documentation.
You can create empty list by []. In order to add new item use append. For add other list use extend.
x = [1, 2, 3]
x.append(4)
x.extend([5, 6])
print(x)
# [1, 2, 3, 4, 5, 6]
if you know the number of columns before hand:
>>> xs = np.array([[1,2,3,4,5],[10,20,30,40,50]])
>>> ys = np.array([], dtype=np.int64).reshape(0,5)
>>> ys
array([], shape=(0, 5), dtype=int64)
>>> np.vstack([ys, xs])
array([[ 1., 2., 3., 4., 5.],
[ 10., 20., 30., 40., 50.]])
if not:
>>> ys = np.array([])
>>> ys = np.vstack([ys, xs]) if ys.size else xs
array([[ 1, 2, 3, 4, 5],
[10, 20, 30, 40, 50]])
In Python, if possible to work with the individual vectors, to append you should use list.append()
>>> E = []
>>> B = np.array([1,2,3,4,5])
>>> C = np.array([10,20,30,40,50])
>>> E = E.append(B)
>>> E = E.append(C)
[array([1, 2, 3, 4, 5]), array([10, 20, 30, 40, 50])]
and then after all append operations are done, return to np.array thusly
>>> E = np.array(E)
array([[ 1, 2, 3, 4, 5],
[10, 20, 30, 40, 50]])
Appending to numpy arrays is computationally expensive in the long run since numpy needs to allocate new array (memory) for the concatenated array, copy the items from the old array to the new array, then finally copy the to be appended array to the new array. I suggest you avoid using np.append. I see that you run into memory issues using list.append, however if you store each ls as numpy arrays you should be able to save some space:
arr = []
ls = np.array([1,2,3])
arr.append(ls)
ls = np.array([1,2,3])
arr.append(ls)
.
.
.
arr = np.array(arr)
However, you can achieve your desired using np.append if you initalize arr with your first array:
arr = np.array([1,2,3,7])
ls = [0,1,0,1]
arr = np.stack([arr, ls])
print(arr[0])
#[1,2,3,7]
print(arr[1])
#[0,1,0,1]
If you do know the upper bound N of items you want in the array, and that you have same sized arrays of length d, you can pre-initialize memory. For example if N=2 and d=3:
N = 2
d = 3
arr = np.empty((N,d))
arr[0] = [1,2,3]
arr[1] = [4,5,6]
print(arr[0])
#[[1, 2, 3],
# [4, 5, 6]]
With a small example like this, examining the whole array will be more instructive than arr[0].
In [1]: arr = np.array([])
...: ls = [1,2,3,7]
...: arr1 = np.append(arr,ls)
In [2]: arr1
Out[2]: array([1., 2., 3., 7.])
The np.append action is just:
In [3]: np.concatenate([arr, np.array(ls)])
Out[3]: array([1., 2., 3., 7.])
It's just an alternative way of calling concatenate. It make some sense when adding one value to an existing array, as in np.append(np.array([1,2,3]), 4). Otherwise it is more of a problem than a solution. It is not a list append clone.
np.array([]) is not a clone of [] either. But your use of that plus the arr=np.append(arr,ls) assignment, suggests you are trying to clone the common list append loop.
Many other SO stress that repeated use of np.append (or anything in the np.concatenate family) is inefficient. Each use requires a full copy. At least temporarily you'll have the original arr and the new one, which is a bit bigger.
You say that building a list of arrays, and doing one np.concatenate (or variant) at the end will push your memory limits. As best I can reason the repeated concatenates will do the same. However, I haven't tried to test this; I don't like to intentionally create a memory error.
Here's a way of doing what you initially tried to do with append:
In [4]: arr = np.array([None,None],dtype=object)
In [5]: arr[0] = [1,2,3,7]
In [6]: arr[1] = np.array([1,2,3,7])
In [7]: arr
Out[7]: array([list([1, 2, 3, 7]), array([1, 2, 3, 7])], dtype=object)
In [8]: arr[0]
Out[8]: [1, 2, 3, 7]
An object dtype array is much closer to a list in nature. It too stores references, rather than values. But it doesn't have any of the computational benefits of a numeric dtype array.
You also tried:
In [11]: arr = np.zeros((0,1))
In [12]: arr
Out[12]: array([], shape=(0, 1), dtype=float64)
In [13]: np.append(arr, [1,2,3,7])
Out[13]: array([1., 2., 3., 7.])
Same thing as the original, except the starting array is different. But np.append docs, if we take time to read them, tell us that it ravels the inputs - unless we provide an axis. np.ravel(arr) is just np.array([])!
But if we just throw in an axis value like 0, we get concatenate dimension errors:
In [19]: np.append(arr, [1,2,3,7],axis=0)
Traceback (most recent call last):
File "<ipython-input-19-a30d55b63eca>", line 1, in <module>
np.append(arr, [1,2,3,7],axis=0)
File "<__array_function__ internals>", line 5, in append
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py", line 4745, in append
return concatenate((arr, values), axis=axis)
File "<__array_function__ internals>", line 5, in concatenate
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 1 has 1 dimension(s)
This traceback shows how the append is translated into a concatenate.
If we want to add a 4 element array to a "empty" we have to start with the right shape:
In [21]: arr = np.zeros((0,4))
In [22]: np.vstack([arr,[1,2,3,7]])
Out[22]: array([[1., 2., 3., 7.]])
vstack is another concatenate front end, one that will turn the [1,2,3,7] into a np.array([[1,2,3,7]]), with shape (1,4). Now a (0,4) can be joined with a (1,4) to make a (1,4).
So not only is repeated concatenate slower, it requires that we get the initial array just right. Despite its superficial similarity to list append, np.append does not allow us to ignore array dimensions and what np.concatenate requires.