You could preallocate the array before assigning the respective values:
a = np.empty(shape=(25, 2), dtype=int)
for x in range(1, 6):
for y in range(1, 6):
index = (x-1)*5+(y-1)
a[index] = x, y
Answer from aasoo on Stack OverflowYou could preallocate the array before assigning the respective values:
a = np.empty(shape=(25, 2), dtype=int)
for x in range(1, 6):
for y in range(1, 6):
index = (x-1)*5+(y-1)
a[index] = x, y
Did you had a look at numpy.ndindex? This could do the trick:
a = np.ndindex(6,6)
You could have some more information on Is there a Python equivalent of range(n) for multidimensional ranges?
python - Construct an numpy array inside a for loop - Stack Overflow
python - creating numpy arrays in a for loop - Stack Overflow
python - how create numpy array with for loop - Stack Overflow
python - Most efficient and most pythonic way to create a NumPy array within a loop - Stack Overflow
Can someone help me build a numpy array with a for loop? This would fit into a for loop of something else I'm doing with for loops which outputs something into each row.
Basically I'd like to build a numpy array that will output into a csv file like this:
1st iteration:
a,b,c,d
2nd iteration:
a,b,c,d
e,f,g,h
3rd iteration:
a,b,c,d
e,f,g,h
x,y,z,q
Thanks a lot! Sorry if my description isn't clear.
NumPy arrays have fixed sizes, so unless you know the size upfront you have to use something that can change sizes, like python lists.
import numpy as np
images = []
for item in dataset:
for row in item["data"]:
images.append(row["sub_data"]) # Add to list
images = np.array(images) # Convert list to np.array()
In order to do this you would either need to use a datatype that's size can be mutated as I did in my other answer or you could also figure out how many images you have before defining the array. (As suggested by @P.Camilleri)
Here's an example of that:
# Count nuber of images
idx_count = 0
for item in dataset:
idx_count += len(item['data'])
# Create an empty numpy array that's Nx3x3
images = np.empty((count, 3, 3))
# Populate numpy array with images
idx = 0
for item in dataset:
for row in item["data"]:
images[idx] = row["sub_data"]
idx += 1
print(images)
This has the advantage that you only allocate the space once, as apposed to using a python list where it's first added to the list then copied to a numpy array.
However, this is at the cost of having to iterate over the data twice.
(Note: Two separate answers so they can be rated separately as I'm not sure which solution is better.)
You can store them in a list:
data = list()
for i in arange(num):
data.append(np.loadtxt("datafromafile%d.txt" % i))
Then you can access each array with:
>>> data[0] + data[1] # sum of the first and second numpy array
As a oneliner it would be:
NUM = 50
data = [np.loadtxt("datafromafile%d.txt" % value) for value in np.arange(NUM)]
or
FILES = ['file1', 'file2', 'file3']
data = {key: np.loadtxt(key) for key in FILES}
as a dict with the filename as key.
Here is the final code, np.fromiter() does the trick and allows to reduce the number of lines by using list comprehension
df = pd.read_excel(path)
print(df.columns)
It outputs ['x1', 'x2']
vars_outputs = ["x1", "x2", "ratio_x1_x2"]
missing_labels = {"ratio_x1_x2" : ["x1", "x2"]}
it = [df[missing_labels[var][0]].iloc[-1]/df[missing_labels[var][1]].iloc[-1] if var in missing_labels
else df[var].iloc[-1] for var in vars_outputs]
t = np.fromiter(it, dtype = float)
Thanks @hpaulj, that might be very useful for me in future. I wasn't aware of the speed up using fromiter()
import timeit
setup = '''
import numpy as np
H, W = 400, 400
it = [(1 + 1 / (i + 0.5)) ** 2 for i in range(W) for j in range(H)]'''
fns = ['''
x = np.array([[(1 + 1 / (i + 0.5)) ** 2 for i in range(W)] for j in range(H)])
''', '''
x = np.fromiter(it, np.float)
x.reshape(H, W)
''']
for f in fns:
print(timeit.timeit(f,setup=setup, number=100))
# gives me
# 6.905218548999983
# 0.5763416080008028
EDIT PS your for loop could be some kind of iterator like
it = [my_df[missing_labels[var][0]][-1]
/ my_df[missing_labels[var][1]][-1] if var in missing_labels
else my_df[var][-1] for var in var_outputs]
There is no need to use a loop. You can use numpy.arange([start, ]stop, [step, ]) to generate a range of numbers.
In your case:
predicted_value = np.arange(9, 33) # Note the 33 if you want 9..32
If you really want to use a loop, there is the option of using a list comprehension:
predicted_value = np.array([i for i in range(9, 33)])
Or an explicit loop, which would be most horrible:
predicted_value = np.empty(33 - 9)
for k, i in enumerate(range(9, 33)):
predicted_value[k] = i
predicted_value = np.array([i for i in range(9, 33)])
itertools would be very convenient for this. E.g.
(modified this answer, also addressing follow up questions)
import numpy as np
import itertools
n_list, m_list, rho_list = [100,1000], [2,10,100], [0.0,0.5,0.9]
f1, f2 = lambda mat: 1, lambda mat: 2 # change accordingly
def f(n, m, rho, f1, f2):
distance_matrix = np.zeros((2, 2))
## replace accordingly
return f1(distance_matrix), f2(distance_matrix)
ff = lambda n, m, rho: f(n, m, rho, f1, f2)
values = np.array([[n, m, rho, *ff(n, m, rho)] for n, m, rho in itertools.product(n_list, m_list, rho_list)])
The *ff(...) is to unpack the tuple into two separate values.
You can do in the following way:
np.array([[n,m,roh,1,2] for n in [100,1000] for m in [2,10,100] for roh in [0.0,0.5,0.9]])
Cheers.
np.column_stack can do the trick:
>>> a = np.array(range(100))
>>> b = np.column_stack((a, a**a))
>>> b
array([[ 0, 1],
[ 1, 1],
[ 2, 4],
[ 3, 27],
[ 4, 256],
[ 5, 3125],
[ 6, 46656],
[ 7, 823543],
[ 8, 16777216],
[ 9, 387420489],
[ 10, 10000000000],
[ 11, 285311670611],
[ 12, 8916100448256],
[ 13, 302875106592253],
A list comprehension could do that for you:
import numpy as np
results = np.array([[x, x**2] for x in range(100)])
That gives you an array of two columns:
Out[5]:
array([[ 0, 0],
[ 1, 1],
[ 2, 4],
[ 3, 9],
...
You've just commented that the function is more complex than just x**2 - one solution is simply to define it as a function, e.g.
import numpy as np
def func(x): # example functionality
y = x**2
y = y*2
return y
results = np.array([[x, func(x)] for x in range(100)])
Out[13]:
array([[ 0, 0],
[ 1, 2],
[ 2, 8],
[ 3, 18],
...
A numpy array must be created with a fixed size. You can create a small one (e.g., one row) and then append rows one at a time, but that will be inefficient. There is no way to efficiently grow a numpy array gradually to an undetermined size. You need to decide ahead of time what size you want it to be, or accept that your code will be inefficient. Depending on the format of your data, you can possibly use something like numpy.loadtxt or various functions in pandas to read it in.
Use a list of 1D numpy arrays, or a list of lists, and then convert it to a numpy 2D array (or use more nesting and get more dimensions if you need to).
import numpy as np
a = []
for i in range(5):
a.append(np.array([1,2,3])) # or a.append([1,2,3])
a = np.asarray(a) # a list of 1D arrays (or lists) becomes a 2D array
print(a.shape)
print(a)
Do you want this?
a = [1,2,3];
b = [4,5,6];
c = [a,b];
c[1][1] # Gives you 5
To do it in a loop
for z in [a,b]:
c.append(z)
# continue as usual......
Also, you don't really need numpy to do this. If you do, follow @Taha s answer above.
c[1] would give [[4 5 6]] to access to 5 you should c[1][0][1]
import numpy as np
b=[4,5,6]
a=[1,2,3]
c=np.array([[a],[b]])
print c[1][0][1]
UPDATE
It's easier to do it this way:
import numpy as np
b=[4,5,6]
a=[1,2,3,9] #I added an element to clarify how to manage indexes in case u have diff sizes
c=np.array([a,b])
#j= sum(1 for x in c if isinstance(x, np.ndarray)) Another way to see how many item in the list
for j in range(len(c)): #Selecting the list (a,b ..)
i=0
while i<=len(c[j])-1: #Looping in the list
print "index (",i,",",j,"):",c[j][i]
i+=1
NumPy provides a 'fromiter' method:
def myfunc(n):
for i in range(n):
yield i**2
np.fromiter(myfunc(5), dtype=int)
which yields
array([ 0, 1, 4, 9, 16])
The recommended way to do this is to preallocate before the loop and use slicing and indexing to insert
my_array = numpy.zeros(1,1000)
for i in xrange(1000):
#for 1D array
my_array[i] = functionToGetValue(i)
#OR to fill an entire row
my_array[i:] = functionToGetValue(i)
#or to fill an entire column
my_array[:,i] = functionToGetValue(i)
numpy does provide an array.resize() method, but this will be far slower due to the cost of reallocating memory inside a loop. If you must have flexibility, then I'm afraid the only way is to create an array from a list.
EDIT: If you are worried that you're allocating too much memory for your data, I'd use the method above to over-allocate and then when the loop is done, lop off the unused bits of the array using array.resize(). This will be far, far faster than constantly reallocating the array inside the loop.
EDIT: In response to @user248237's comment, assuming you know any one dimension of the array (for simplicity's sake):
my_array = numpy.array(10000, SOMECONSTANT)
for i in xrange(someVariable):
if i >= my_array.shape[0]:
my_array.resize((my_array.shape[0]*2, SOMECONSTANT))
my_array[i:] = someFunction()
#lop off extra bits with resize() here
The general principle is "allocate more than you think you'll need, and if things change, resize the array as few times as possible". Doubling the size could be thought of as excessive, but in fact this is the method used by several data structures in several standard libraries in other languages (java.util.Vector does this by default for example. I think several implementations of std::vector in C++ do this as well).