You could preallocate the array before assigning the respective values:
a = np.empty(shape=(25, 2), dtype=int)
for x in range(1, 6):
for y in range(1, 6):
index = (x-1)*5+(y-1)
a[index] = x, y
Answer from aasoo on Stack OverflowYou could preallocate the array before assigning the respective values:
a = np.empty(shape=(25, 2), dtype=int)
for x in range(1, 6):
for y in range(1, 6):
index = (x-1)*5+(y-1)
a[index] = x, y
Did you had a look at numpy.ndindex? This could do the trick:
a = np.ndindex(6,6)
You could have some more information on Is there a Python equivalent of range(n) for multidimensional ranges?
python - Construct an numpy array inside a for loop - Stack Overflow
python - Most efficient and most pythonic way to create a NumPy array within a loop - Stack Overflow
python - creating numpy arrays in a for loop - Stack Overflow
python - how create numpy array with for loop - Stack Overflow
Can someone help me build a numpy array with a for loop? This would fit into a for loop of something else I'm doing with for loops which outputs something into each row.
Basically I'd like to build a numpy array that will output into a csv file like this:
1st iteration:
a,b,c,d
2nd iteration:
a,b,c,d
e,f,g,h
3rd iteration:
a,b,c,d
e,f,g,h
x,y,z,q
Thanks a lot! Sorry if my description isn't clear.
NumPy arrays have fixed sizes, so unless you know the size upfront you have to use something that can change sizes, like python lists.
import numpy as np
images = []
for item in dataset:
for row in item["data"]:
images.append(row["sub_data"]) # Add to list
images = np.array(images) # Convert list to np.array()
In order to do this you would either need to use a datatype that's size can be mutated as I did in my other answer or you could also figure out how many images you have before defining the array. (As suggested by @P.Camilleri)
Here's an example of that:
# Count nuber of images
idx_count = 0
for item in dataset:
idx_count += len(item['data'])
# Create an empty numpy array that's Nx3x3
images = np.empty((count, 3, 3))
# Populate numpy array with images
idx = 0
for item in dataset:
for row in item["data"]:
images[idx] = row["sub_data"]
idx += 1
print(images)
This has the advantage that you only allocate the space once, as apposed to using a python list where it's first added to the list then copied to a numpy array.
However, this is at the cost of having to iterate over the data twice.
(Note: Two separate answers so they can be rated separately as I'm not sure which solution is better.)
Here is the final code, np.fromiter() does the trick and allows to reduce the number of lines by using list comprehension
df = pd.read_excel(path)
print(df.columns)
It outputs ['x1', 'x2']
vars_outputs = ["x1", "x2", "ratio_x1_x2"]
missing_labels = {"ratio_x1_x2" : ["x1", "x2"]}
it = [df[missing_labels[var][0]].iloc[-1]/df[missing_labels[var][1]].iloc[-1] if var in missing_labels
else df[var].iloc[-1] for var in vars_outputs]
t = np.fromiter(it, dtype = float)
Thanks @hpaulj, that might be very useful for me in future. I wasn't aware of the speed up using fromiter()
import timeit
setup = '''
import numpy as np
H, W = 400, 400
it = [(1 + 1 / (i + 0.5)) ** 2 for i in range(W) for j in range(H)]'''
fns = ['''
x = np.array([[(1 + 1 / (i + 0.5)) ** 2 for i in range(W)] for j in range(H)])
''', '''
x = np.fromiter(it, np.float)
x.reshape(H, W)
''']
for f in fns:
print(timeit.timeit(f,setup=setup, number=100))
# gives me
# 6.905218548999983
# 0.5763416080008028
EDIT PS your for loop could be some kind of iterator like
it = [my_df[missing_labels[var][0]][-1]
/ my_df[missing_labels[var][1]][-1] if var in missing_labels
else my_df[var][-1] for var in var_outputs]
You can store them in a list:
data = list()
for i in arange(num):
data.append(np.loadtxt("datafromafile%d.txt" % i))
Then you can access each array with:
>>> data[0] + data[1] # sum of the first and second numpy array
As a oneliner it would be:
NUM = 50
data = [np.loadtxt("datafromafile%d.txt" % value) for value in np.arange(NUM)]
or
FILES = ['file1', 'file2', 'file3']
data = {key: np.loadtxt(key) for key in FILES}
as a dict with the filename as key.
NumPy provides a 'fromiter' method:
def myfunc(n):
for i in range(n):
yield i**2
np.fromiter(myfunc(5), dtype=int)
which yields
array([ 0, 1, 4, 9, 16])
The recommended way to do this is to preallocate before the loop and use slicing and indexing to insert
my_array = numpy.zeros(1,1000)
for i in xrange(1000):
#for 1D array
my_array[i] = functionToGetValue(i)
#OR to fill an entire row
my_array[i:] = functionToGetValue(i)
#or to fill an entire column
my_array[:,i] = functionToGetValue(i)
numpy does provide an array.resize() method, but this will be far slower due to the cost of reallocating memory inside a loop. If you must have flexibility, then I'm afraid the only way is to create an array from a list.
EDIT: If you are worried that you're allocating too much memory for your data, I'd use the method above to over-allocate and then when the loop is done, lop off the unused bits of the array using array.resize(). This will be far, far faster than constantly reallocating the array inside the loop.
EDIT: In response to @user248237's comment, assuming you know any one dimension of the array (for simplicity's sake):
my_array = numpy.array(10000, SOMECONSTANT)
for i in xrange(someVariable):
if i >= my_array.shape[0]:
my_array.resize((my_array.shape[0]*2, SOMECONSTANT))
my_array[i:] = someFunction()
#lop off extra bits with resize() here
The general principle is "allocate more than you think you'll need, and if things change, resize the array as few times as possible". Doubling the size could be thought of as excessive, but in fact this is the method used by several data structures in several standard libraries in other languages (java.util.Vector does this by default for example. I think several implementations of std::vector in C++ do this as well).
np.column_stack can do the trick:
>>> a = np.array(range(100))
>>> b = np.column_stack((a, a**a))
>>> b
array([[ 0, 1],
[ 1, 1],
[ 2, 4],
[ 3, 27],
[ 4, 256],
[ 5, 3125],
[ 6, 46656],
[ 7, 823543],
[ 8, 16777216],
[ 9, 387420489],
[ 10, 10000000000],
[ 11, 285311670611],
[ 12, 8916100448256],
[ 13, 302875106592253],
A list comprehension could do that for you:
import numpy as np
results = np.array([[x, x**2] for x in range(100)])
That gives you an array of two columns:
Out[5]:
array([[ 0, 0],
[ 1, 1],
[ 2, 4],
[ 3, 9],
...
You've just commented that the function is more complex than just x**2 - one solution is simply to define it as a function, e.g.
import numpy as np
def func(x): # example functionality
y = x**2
y = y*2
return y
results = np.array([[x, func(x)] for x in range(100)])
Out[13]:
array([[ 0, 0],
[ 1, 2],
[ 2, 8],
[ 3, 18],
...
Do you want this?
a = [1,2,3];
b = [4,5,6];
c = [a,b];
c[1][1] # Gives you 5
To do it in a loop
for z in [a,b]:
c.append(z)
# continue as usual......
Also, you don't really need numpy to do this. If you do, follow @Taha s answer above.
c[1] would give [[4 5 6]] to access to 5 you should c[1][0][1]
import numpy as np
b=[4,5,6]
a=[1,2,3]
c=np.array([[a],[b]])
print c[1][0][1]
UPDATE
It's easier to do it this way:
import numpy as np
b=[4,5,6]
a=[1,2,3,9] #I added an element to clarify how to manage indexes in case u have diff sizes
c=np.array([a,b])
#j= sum(1 for x in c if isinstance(x, np.ndarray)) Another way to see how many item in the list
for j in range(len(c)): #Selecting the list (a,b ..)
i=0
while i<=len(c[j])-1: #Looping in the list
print "index (",i,",",j,"):",c[j][i]
i+=1
I have a project were I need to do something very similar. You can implement dynamic resizing in your loop, but python's list type is actually implemented as a dynamic array so you might as well take advantage of the tools that are already available to you. You can do something like this:
delta_Array = np.array([0.01,0.02,0.03, 0.04, 0.05, 0.06,0.07, 0.08, 0.09, 0.10])
theta_Matrix = []
for i in range(N):
t = Ridge(Xtrain, ytrain, .3)
theta_Matrix.append(t)
theta_Matrix = np.array(theta_Matrix)
I should mention that if you already know the size you expect for theta_Matrix, you'll get the best performance by doing something like:
delta_Array = np.array([0.01,0.02,0.03, 0.04, 0.05, 0.06,0.07, 0.08, 0.09, 0.10])
theta_Matrix = np.zeros((N, 8))
for i in range(N):
t = Ridge(Xtrain, ytrain, .3)
theta_Matrix[i] = t
np.append returns the concatenated array, and you're ignoring its return value. You should also consider using np.vstack instead, since that stacks row vectors into matrices (append can do it but it takes extra arguments).
However, running np.append or np.vstack in a loop is still not a very good idea as constructing the matrix will take quadratic time. It's better to preallocate an array and then fill it row by row using slicing. If you don't know how large it will need to be, consider iteratively doubling its size (see Wikipedia, Dynamic array) with np.resize.
numpy.zerosReturn a new array of given shape and type, filled with zeros.
or
numpy.onesReturn a new array of given shape and type, filled with ones.
or
numpy.emptyReturn a new array of given shape and type, without initializing entries.
However, the mentality in which we construct an array by appending elements to a list is not much used in numpy, because it's less efficient (numpy datatypes are much closer to the underlying C arrays). Instead, you should preallocate the array to the size that you need it to be, and then fill in the rows. You can use numpy.append if you must, though.
The way I usually do that is by creating a regular list, then append my stuff into it, and finally transform the list to a numpy array as follows :
Use with AIimport numpy as np
big_array = [] # empty regular list
for i in range(5):
arr = i*np.ones((2,4)) # for instance
big_array.append(arr)
big_np_array = np.array(big_array) # transformed to a numpy array
of course your final object takes twice the space in the memory at the creation step, but appending on python list is very fast, and creation using np.array() also.
There is no need to use a loop. You can use numpy.arange([start, ]stop, [step, ]) to generate a range of numbers.
In your case:
predicted_value = np.arange(9, 33) # Note the 33 if you want 9..32
If you really want to use a loop, there is the option of using a list comprehension:
predicted_value = np.array([i for i in range(9, 33)])
Or an explicit loop, which would be most horrible:
predicted_value = np.empty(33 - 9)
for k, i in enumerate(range(9, 33)):
predicted_value[k] = i
predicted_value = np.array([i for i in range(9, 33)])
A numpy array must be created with a fixed size. You can create a small one (e.g., one row) and then append rows one at a time, but that will be inefficient. There is no way to efficiently grow a numpy array gradually to an undetermined size. You need to decide ahead of time what size you want it to be, or accept that your code will be inefficient. Depending on the format of your data, you can possibly use something like numpy.loadtxt or various functions in pandas to read it in.
Use a list of 1D numpy arrays, or a list of lists, and then convert it to a numpy 2D array (or use more nesting and get more dimensions if you need to).
import numpy as np
a = []
for i in range(5):
a.append(np.array([1,2,3])) # or a.append([1,2,3])
a = np.asarray(a) # a list of 1D arrays (or lists) becomes a 2D array
print(a.shape)
print(a)