First you have to install numpy using
$ pip install numpy
Then the following should work
import numpy as np
n = 100
matrix = np.zeros((n,2)) # Pre-allocate matrix
for i in range(1,n):
matrix[i,:] = [3*i, i**2]
A faster alternative:
col1 = np.arange(3,3*n,3)
col2 = np.arange(1,n)
matrix = np.hstack((col1.reshape(n-1,1), col2.reshape(n-1,1)))
Even faster, as Divakar suggested
I = np.arange(n)
matrix = np.column_stack((3*I, I**2))
First you have to install numpy using
$ pip install numpy
Then the following should work
import numpy as np
n = 100
matrix = np.zeros((n,2)) # Pre-allocate matrix
for i in range(1,n):
matrix[i,:] = [3*i, i**2]
A faster alternative:
col1 = np.arange(3,3*n,3)
col2 = np.arange(1,n)
matrix = np.hstack((col1.reshape(n-1,1), col2.reshape(n-1,1)))
Even faster, as Divakar suggested
I = np.arange(n)
matrix = np.column_stack((3*I, I**2))
This is very pythonic form to produce a list, which you can easily swap e.g. for np.array, set, generator etc.
n = 10
[[i*3, i**2] for i, i in zip(range(0,n), range(0,n))]
If you want to add another column it's no problem. Simply
[[i*3, i**2, i**(0.5)] for i, i in zip(range(0,n), range(0,n))]
How to fill a matrix using numpy
python - Fill empty numpy array inside for loop - Stack Overflow
python - How do i fill arrays with loops? - Stack Overflow
python - How to declare and fill an array in NumPy? - Stack Overflow
Hello everyone,
I haven't done any type of linear algebra work with python before, only matlab. In matlab you create a matrix with the proper dimensions, then start filling it up with vectors of the appropriate size. I've tried to look up how to do this in python using a loop, and it doesn't seem as straightforward. It seems you first need to create a list, and append that list. Then convert that list to a numpy array. And then from there it appears you stack them? But stacking seems to only stack 1D and 2D arrays, so it's not exactly what I'm looking for.
For example:
import numpy as np
matrix=np.zeros((rows,columns))
list=[]
for i in range(columns):
for x in range(rows):
list.append(x)
arr=np.array(list)
matrix.append(arr)
list.clear() So in the above case, I've made a list, and converted it into an array with shape (rows, 1). Then I want to add it to the first column of my matrix. Then through the loop, I start to fill my matrix up by filling it one by one with these vectors. However, the issue is I'm trying to add a 1D array, to a multi-D matrix (which numpy doesn't allow you to do).
Hpw do I fill up a matrix using numpy?
You don't really need to fill the array. np.linspace generates a uniformly spaced numpy array for you. However, given your requirement you can do this:
Solution 1: Loop method
import numpy as np
from numpy import log
#filling zeros
x_array = np.zeros(101)
y_array = np.zeros(101)
#generating uniformly spaced numbers
x=np.linspace(1,10,101)
#filling the array with loop
for i in range(0,len(x)):
x_array[i] = x[i] #filling x_array
y_array[i] = log(x[i]) #filling y_array
print(x_array,y_array)
Solution 2: List comprehension method
import numpy as np
from numpy import log
#filling zeros
x_array = np.zeros(101)
y_array = np.zeros(101)
#generating uniformly spaced numbers
x=np.linspace(1,10,101)
#filling the array with list comprehension
x_array = [ x[i] for i in range(0,len(x))]
y_array = [ log(x[i]) for i in range(0,len(x))]
print(len(x_array),len(y_array))
print(x_array,y_array)
Output:
#length
101 101
#x_array
[1.0, 1.09, 1.18, 1.27, 1.3599999999999999, 1.45, 1.54, 1.63, 1.72, 1.81, 1.9, 1.99, 2.08, 2.17, 2.26, 2.3499999999999996, 2.44, 2.5300000000000002, 2.62, 2.71, 2.8, 2.8899999999999997, 2.98, 3.07, 3.16, 3.25, 3.34, 3.4299999999999997, 3.52, 3.61, 3.6999999999999997, 3.79, 3.88, 3.9699999999999998, 4.0600000000000005, 4.15, 4.24, 4.33, 4.42, 4.51, 4.6, 4.6899999999999995, 4.779999999999999, 4.869999999999999, 4.96, 5.05, 5.14, 5.2299999999999995, 5.32, 5.41, 5.5, 5.59, 5.68, 5.77, 5.859999999999999, 5.95, 6.04, 6.13, 6.22, 6.31, 6.3999999999999995, 6.49, 6.58, 6.67, 6.76, 6.85, 6.9399999999999995, 7.029999999999999, 7.12, 7.21, 7.3, 7.39, 7.4799999999999995, 7.569999999999999, 7.66, 7.75, 7.84, 7.93, 8.02, 8.11, 8.2, 8.29, 8.379999999999999, 8.469999999999999, 8.559999999999999, 8.649999999999999, 8.739999999999998, 8.83, 8.92, 9.01, 9.1, 9.19, 9.28, 9.37, 9.459999999999999, 9.549999999999999, 9.64, 9.73, 9.82, 9.91, 10.0]
#y_erray
[0.0, 0.08617769624105241, 0.16551443847757333, 0.23901690047049992, 0.30748469974796055, 0.371563556432483, 0.43178241642553783, 0.4885800148186709, 0.5423242908253617, 0.5933268452777344, 0.6418538861723947, 0.688134638736401, 0.7323678937132266, 0.7747271675523681, 0.8153648132841944, 0.8544153281560675, 0.8919980393051105, 0.928219302739429, 0.9631743177730056, 0.9969486348916096, 1.0296194171811581, 1.0612565021243408, 1.091923300517313, 1.1216775615991057, 1.1505720275988207, 1.1786549963416462, 1.205970806988609, 1.2325602611778486, 1.2584609896100056, 1.2837077723447896, 1.3083328196501787, 1.332366019094335, 1.355835153635182, 1.378766094699099, 1.4011829736136414, 1.423108334242607, 1.4445632692438664, 1.4655675420143985, 1.4861396960896067, 1.506297153514587, 1.5260563034950492, 1.5454325824581878, 1.5644405465033644, 1.5830939370944983, 1.601405740736836, 1.6193882432872684, 1.6370530794670737, 1.6544112780768314, 1.6714733033535532, 1.6882490928583902, 1.7047480922384253, 1.7209792871670078, 1.7369512327330598, 1.7526720805200082, 1.7681496035889213, 1.7833912195575383, 1.7984040119467235, 1.81319474994812, 1.827769906751088, 1.8421356765531218, 1.856297990365626, 1.8702625307159986, 1.884034745337226, 1.8976198599275322, 1.9110228900548727, 1.9242486522741338, 1.937301774518713, 1.9501867058225735, 1.9629077254238845, 1.9754689512968577, 1.9878743481543455, 2.0001277349601105, 2.012232791986386, 2.024193067449357, 2.0360119837525, 2.0476928433652555, 2.0592388343623163, 2.0706530356467567, 2.081938421878423, 2.0930978681273213, 2.1041341542702074, 2.1150499691472033, 2.1258479144939915, 2.136530508663963, 2.1471001901536506, 2.157559320943788, 2.167910189667444, 2.178155014615869, 2.188295946591918, 2.1983350716202463, 2.2082744135228043, 2.2181159363675955, 2.2278615467981093, 2.2375130962503307, 2.2470723830637866, 2.2565411544926386, 2.265921108622454, 2.2752138961979136, 2.2844211223663744, 2.2935443483418965, 2.302585092994046]
EDIT:
Adding the plotting functionality:
#add this code below the above code if you want a plot
import matplotlib.pyplot as plt
plt.plot(x_array,y_array)
plt.title(r'
')
plt.show()
Output:
You can generate x_array using range() and then fill y_array accordingly using math.log():
from math import log
x_array = list(range(1, 11))
y_array = []
for val in x_array:
y_array.append(log(val))
print(x_array)
print(y_array)
Or, a bit concisely, use list-comprehension:
from math import log
x_array = list(range(1, 11))
y_array = [log(val) for val in x_array]
print(x_array)
print(y_array)
You could do:
data1 = np.zeros((56,4))
to get a 56 by 4 array. If you don't like to start the array with 0, you could use np.ones or np.empty or np.ones((56, 4)) * np.nan
Then, in most cases it is best not to python-loop if not needed for performance reasons. So as an example this would do your loop:
data[:, 3] = 32
data1 = np.array([ra,dec,[32]*len(ra)])
Gives a single-line solution to your problem; but for efficiency, allocating an empty array first and then copying in the relevant parts would be preferable, so you avoid the construction of the dummy list.
NumPy 1.8 introduced np.full(), which is a more direct method than empty() followed by fill() for creating an array filled with a certain value:
>>> np.full((3, 5), 7)
array([[ 7., 7., 7., 7., 7.],
[ 7., 7., 7., 7., 7.],
[ 7., 7., 7., 7., 7.]])
>>> np.full((3, 5), 7, dtype=int)
array([[7, 7, 7, 7, 7],
[7, 7, 7, 7, 7],
[7, 7, 7, 7, 7]])
This is arguably the way of creating an array filled with certain values, because it explicitly describes what is being achieved (and it can in principle be very efficient since it performs a very specific task).
Updated for Numpy 1.7.0:(Hat-tip to @Rolf Bartstra.)
a=np.empty(n); a.fill(5) is fastest.
In descending speed order:
%timeit a=np.empty(10000); a.fill(5)
100000 loops, best of 3: 5.85 us per loop
%timeit a=np.empty(10000); a[:]=5
100000 loops, best of 3: 7.15 us per loop
%timeit a=np.ones(10000)*5
10000 loops, best of 3: 22.9 us per loop
%timeit a=np.repeat(5,(10000))
10000 loops, best of 3: 81.7 us per loop
%timeit a=np.tile(5,[10000])
10000 loops, best of 3: 82.9 us per loop
You seem to be updating your result array one column at a time, using data from the previously updated column. That makes it hard to vectorize the outer loop, I don't think you can do it unless there is some structure to your data that can be exploited. The inner loop is very straightforward to vectorize:
for i in xrange(1, d):
j = np.arange(k-i-1, k+1)
result[j, i] = ((x - y[j]) / (y[j+i] - y[j]) * result[j, i-1] +
(y[j+i+1] - x) / (y[j+i+1] - y[j+1]) * result[j+1, i-1])
There is some marginal improvement you could get by defining a base_j = np.arange(k-d-2, k+1) array outside the loop, and then slicing it inside the loop with something like j = base_j[d-i+1:].
You can get rid of the inner loop via clever broadcasting
tmp = ((x - y[k-i+1:k+1])/(y[k+1:k+1+i] - y[j]))*result[k-i+1:k+1,i-1] + ((y[k+2:k+i+2]-x)/(y[k+2:k+i+2] - y[k-i+2:k+2]))*result[k-i+2:k+2,i-1]
result[k-i+1:k+1, i] = tmp
but because you bounds in the inner loop depend on the outer-loop you can not remove it through broadcasting as well.
I used tmp in an overabundance of caution
tile and repeat are handy functions when you want to repeat an array in various ways:
In [233]: np.tile(np.array([4,6,6,1]),(3,1))
Out[233]:
array([[4, 6, 6, 1],
[4, 6, 6, 1],
[4, 6, 6, 1]])
On the failure, note the docs for fill:
a.fill(value)
Fill the array with a scalar value.
np.array([4,6,6,1]) is not a scalar value. a was initialized as a 3 element float array.
It is possible to assign values to elements of an array, provided the shapes are right:
In [241]: a=np.empty(3)
In [242]: a[:]=np.array([1,2,3]) # 3 numbers into 3 slots
In [243]: a
Out[243]: array([ 1., 2., 3.])
In [244]: a=np.empty((3,4))
In [245]: a[:]=np.array([1,2,3,4]) # 4 numbers into 4 columns
In [246]: a
Out[246]:
array([[ 1., 2., 3., 4.],
[ 1., 2., 3., 4.],
[ 1., 2., 3., 4.]])
This fill works with an object type array, but the result is quite different, and should be used with considerable caution:
In [247]: a=np.empty(3, object)
In [248]: a
Out[248]: array([None, None, None], dtype=object)
In [249]: a.fill(np.array([1,2,3,4]))
In [250]: a
Out[250]: array([array([1, 2, 3, 4]), array([1, 2, 3, 4]), array([1, 2, 3, 4])], dtype=object)
This (3,) array is not the same as the (3,4) array produced by other methods. Each element of the object array is a pointer to the same thing. Changing a value in one element of a changes that value in all the elements (because they are the same object).
In [251]: a[0][3]=5
In [252]: a
Out[252]: array([array([1, 2, 3, 5]), array([1, 2, 3, 5]), array([1, 2, 3, 5])], dtype=object)
Use broadcasting
vstack, tile, and repeat are all great and whatnot, but broadcasting can be several orders of magnitude faster...
import numpy as np
from time import time
t = time()
for _ in xrange(10000):
a = np.array([4,6,6,1])
b = np.vstack((a,)*100)
print time()-t
t = time()
for _ in xrange(10000):
a = np.array([4,6,6,1])
b = np.tile(a,(3,1))
print time()-t
t = time()
for _ in xrange(10000):
a = np.array([4,6,6,1])
b = np.empty([100,a.shape[0]])
b[:] = a
print time()-t
prints:
2.76399993896
0.140000104904
0.0490000247955
If we assume sample_function(x) is a scaler, you can use:
a = np.full(shape=1000, sample_function(x)) # or smth like --> np.full((1, 1000), sample_function(x))
but if you have an array that you would to apply a function on all of the elements:
a = np.apply_along_axis(sample_function, axis, arr) # or --> np.apply_over_axes
You can do:
a = example_function(np.arange(1000))
where example_function is arbitrary, i.e.:
def example_function(x):
return x**2
You can easily do this by using list comprehension.
xrange(0, len(crave), 3) is used to iterate from 0 to len(crave) with an interval of 3. and then we use list slicing crave[i:i+3] to extract the required data.
crave=['word1','word2','word3','word4','word5','word6','word7']
ITEM = [crave[i:i+3] for i in xrange(0, len(crave), 3)]
print ITEM
>>> [['word1', 'word2', 'word3'], ['word4', 'word5', 'word6'], ['word7']]
try to this.
crave = ['word1', 'word2', 'word3', 'word4', 'word5', 'word6', 'word7']
li = []
for x in range(0, len(crave), 3):
li.append(crave[x:x+3])
print li
Hi r/learnpython
Here's the problem: I have some really large data that I need to transform for future use. The data is made of more than one billion "events" (each "event" having lots of associated values).
The events are accessed dynamically by an external library (CERN's ROOT): looping across all of them is not a problem. Currently the code works like that:
nrofevents = mydata.size() # 1 billion
array = numpy.zeros( [nrofevents, 5] ) # Requires 40 Gb of memory!
for i in xrange(nrofevents) :
event = mydata.getEvent()
is_good = do_stuff(event)
if not select: continue # row "i" in array is kept as having only zeroes. I know the "continue" is useless here but that's a simplification of my code
else:
fill( array[i], event )
array = array[numpy.any(array,axis=1)] # At the end of loop, I keep only non-zero rowsThe final array, after call of numpy.any, is much smaller (~2 orders of magnitude) and should not be a memory problem. The problem is instead the allocation of the very large array just before the loop, especially considering that most rows will be sent to trash.
I'm using numpy arrays exactly because they are more memory efficient than python lists.
Do you have any suggestion on how to deal with that?
I see two options
-
Beef up your machine or gain access to a beefy machine.
-
Use a database. If you're worried about IO write times, Database engines are there to make it slightly better. You can also improve performance by focusing on the right data sampling strategies to load a fair subset of your data into main memory. My favorite one would be https://en.wikipedia.org/wiki/Reservoir_sampling/
How about saving the processed data to a file as you loop through your events?