What you show is 'pythonic' in the sense that it uses a Python list and iteration approach. The only use of numpy is in assigning the values, M{i,j] =. Lists don't take that kind of index.

To make most use of numpy, make index grids or arrays, and calculate all values at once, without explicit loop. For example, in your case:

In [333]: N=10
In [334]: I,J = np.ogrid[0:10,0:10]
In [335]: I
Out[335]: 
array([[0],
       [1],
       [2],
       [3],
       [4],
       [5],
       [6],
       [7],
       [8],
       [9]])
In [336]: J
Out[336]: array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])
In [337]: M = 1/((I + 2*J + 1)**2)
In [338]: M
Out[338]: 
array([[ 1.        ,  0.11111111,  0.04      ,  0.02040816,  0.01234568,
         0.00826446,  0.00591716,  0.00444444,  0.00346021,  0.00277008],
 ...
       [ 0.01      ,  0.00694444,  0.00510204,  0.00390625,  0.00308642,
         0.0025    ,  0.00206612,  0.00173611,  0.00147929,  0.00127551]])

ogrid is one of several ways of construction sets of arrays that can be 'broadcast' together. meshgrid is another common function.

In your case, the equation is one that works well with 2 arrays like this. It depends very much on broadcasting rules, which you should study.

If the function only takes scalar inputs, we will have to use some form of iteration. That has been a frequent SO question; search for [numpy] vectorize.

There is a possibility to (ab)use scipy.ndimage.convolve for this kind of problem. Maybe using some filter methods in scipy might work too and be better as it does not rely on scipy.ndimage.convolve working in place (and I could imagine this changing in some far future). (EDIT: first wrote scipy.signal.convolve which like the numpy.convolve, cannot do this)

The trick is that this convolve function can be used in-place so the double for loop:

for i in xrange(1, flux.shape[0]):
    for j in xrange(1, flux.shape[1]):
        flux[i,j] = 0.5*(flux[i-1,j] + bz[i-1,j]*dx) + 0.5*(flux[i,j-1] - bx[i,j-1]*dz)

Can be replaced by (sorry, for the need of so many temporary arrays...):

from scipy.ndimage import convolve
_flux = np.zeros((flux.shape[0]+1, flux.shape[1]+1), dtype=flux.dtype)
temp_bx = np.zeros((bx.shape[0]+1, bx.shape[1]+1), dtype=bx.dtype)
temp_bz = np.zeros((bz.shape[0]+1, bz.shape[1]+1), dtype=bz.dtype)

_flux[:-1,:-1] = flux
convolve(_flux[:-1,:-1], [[0, 0.5], [0.5, 0]], _flux[1:,1:])

temp_bz[1:,1:-1] = bz[:,1:]*dx
temp_bx[1:-1,1:] = bx[1:,:]*dz

conv_b = np.array([[0.0, 0.5], [0.5, 0.5]])
convolve(temp_bz[:-1,:-1], [[0.5, 0.5], [0.5, 0.]], temp_bz[1:,1:])
convolve(temp_bx[:-1,:-1], [[-0.5, 0.5], [0.5, 0.]], temp_bx[1:,1:])

flux = _flux[:-1,:-1] + temp_by[:-1,:-1] + temp_bx[:-1,:-1]

Unfortunatly it means we need to fiddle apart how bx, bz go into the final results, but this approach avoids creating large powers of two and should be noticeably faster then the previous answer as well.

(Note that the numpy convolve function does not allow for this inplace usage.)

You can loop through the arrays and append to the new ones:

array1 = []
array2 = []

for array in my_array:
  for nested_array in array:
    # nested_array is of form [[ 1 10]] here, you need to index it
    # first with [0] then with the element you want to access [0] or [1]
    array1.append(nested_array[0][0])
    array2.append(nested_array[0][1])

You just have to think about the structure of the input data and how to get the values you need in the order that you need.

The output:

>>> array1
[1, 2, 3, 4, 5, 6, 7, 8]
>>> array2
[10, 11, 12, 13, 14, 15, 16, 17]

The numpy array you are defining is not the right shape for the loop you are using. b = np.array([]) gives you an array of shape (0,)

You can use something like np.zeros to define your 2D array.

import numpy as np
b = np.zeros((3,6))
for i in range(2):
    for j in range(5):
        b[i][j]=i+j
print(b)

The output will be

[[0. 1. 2. 3. 4. 0.]
 [1. 2. 3. 4. 5. 0.]
 [0. 0. 0. 0. 0. 0.]]

Another option is to make a 2D list, fill it up in for loops, and convert to numpy array later

import numpy as np

#2D array
b = [ [0 for _ in range(6)] for _ in range(3)]

#Fill up array with for loops
for i in range(2):
    for j in range(5):
        b[i][j]=i+j

#Convert to numpy array
print(np.array(b))

The output will be

[[0 1 2 3 4 0]
 [1 2 3 4 5 0]
 [0 0 0 0 0 0]]

You can use broadcasting for this:

a = np.array([
    [1,1,1,1],
    [2,2,2,2],
    [3,3,3,3]
])
b = np.array([
    [1,1,1,1],
    [2,2,2,2]
])

def euc_distance(array1, array2):
    return np.sqrt(np.sum((array1 - array2)**2, axis = -1))

print(euc_distance(a[None, :, :], b[:, None, :]))
# [[0. 2. 4.]
#  [2. 0. 2.]]

Comparing the times for a dataset of your size:

a = np.random.rand(700000, 4)
b = np.random.rand(4, 4)

c = euc_distance(a[None, :, :], b[:, None, :])
d = np.array([euc_distance(a, val) for val in b])
e = np.array([euc_distance(val, b) for val in a]).T

np.allclose(c, d)
# True
np.allclose(d, e)
# True

%timeit euc_distance(a[None, :, :], b[:, None, :])
# 113 ms ± 4.56 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit np.array([euc_distance(a, val) for val in b])
# 115 ms ± 4.32 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit np.array([euc_distance(val, b) for val in a])
# 7.03 s ± 216 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

With a numpy tag I assume 'optimize' means cast these loops as compiled numpy array expressions. I don't think there's a way to do without first collecting all the data as numpy arrays, which will require the same loops.

You have a list of n variables. Each variable has a list of ? parameters. Each parameter has 3 or 4 attributes.

So

rates = np.array([v.rate for v in variables])
values = np.array([[p.value for p in v.parameters] for v in variables]
index1s = <dito>   (?,n) array
index2s = <dita>   (?,n) array

self.state.num is already a 2d array, size compatible with the range of values in index1s and index2s.

Given those 1 and 2d arrays we should be able to derive mat1 and mat2 with whole array operations. If ? is small relative to m and the range of values in index1 and index2 it might be worth do this. I don't have a realistic feel for your data.

===========

You mention the map function, comprehension expressions, and a bit of itertools and generators. These can make the code look a little cleaner but don't make much difference in speed.

I demonstrated the use of list comprehensions. Those expression can be more elaborate, but often at the cost of readability. I like writing helper functions to hide details. A generator comprehension can replace a list comprehension that feeds another. Maps cover the same functionality.

Since variables and parameters have attributes I assume you have defined them in classes. You could write methods that extract those attributes as simple lists or arrays.

class Variable(....):
   ....
   def get_values(self):
       return [p.value for p in self.parameters]
   def get_rate(self, state):
       rate = self.rate
       for parameter in self.parameters:
            if parameter.value < 0 and                
                state.num[parameter.index1, parameter.index2] <= 0:
            rate = 0
       return rate

values = [v.get_values() for v in variables]
rates = [v.get_rate(self.state) for v in variables]

You could even write these helper function without the class structure.

That doesn't speed up anything; it just hides some details in the object.

You can certainly do with np.einsum:

Z[i,a,b,j] += X[m,e,b,j] * Y[m,e,a,i] * 2

translates to

Z = np.einsum('mebj,meai->iabj', X,Y) * 2

This is a common conceptual difficulty when learning to use NumPy effectively. Normally, data processing in Python is best expressed in terms of iterators, to keep memory usage low, to maximize opportunities for parallelism with the I/O system, and to provide for reuse and combination of parts of algorithms.

But NumPy turns all that inside out: the best approach is to express the algorithm as a sequence of whole-array operations, to minimize the amount of time spent in the slow Python interpreter and maximize the amount of time spent in fast compiled NumPy routines.

Here's the general approach I take:

1. Keep the original version of the function (which you are confident is correct) so that you can test it against your improved versions both for correctness and speed.

2. Work from the inside out: that is, start with the innermost loop and see if can be vectorized; then when you've done that, move out one level and continue.

3. Spend lots of time reading the NumPy documentation. There are a lot of functions and operations in there and they are not always brilliantly named, so it's worth getting to know them. In particular, if you find yourself thinking, "if only there were a function that did such-and-such," then it's well worth spending ten minutes looking for it. It's usually in there somewhere.

There's no substitute for practice, so I'm going to give you some example problems. The goal for each problem is to rewrite the function so that it is fully vectorized: that is, so that it consists of a sequence of NumPy operations on whole arrays, with no native Python loops (no for or while statements, no iterators or comprehensions).

Problem 1

def sumproducts(x, y):
    """Return the sum of x[i] * y[j] for all pairs of indices i, j.

    >>> sumproducts(np.arange(3000), np.arange(3000))
    20236502250000

    """
    result = 0
    for i in range(len(x)):
        for j in range(len(y)):
            result += x[i] * y[j]
    return result

Problem 2

def countlower(x, y):
    """Return the number of pairs i, j such that x[i] < y[j].

    >>> countlower(np.arange(0, 200, 2), np.arange(40, 140))
    4500

    """
    result = 0
    for i in range(len(x)):
        for j in range(len(y)):
            if x[i] < y[j]:
                result += 1
    return result

Problem 3

def cleanup(x, missing=-1, value=0):
    """Return an array that's the same as x, except that where x ==
    missing, it has value instead.

    >>> cleanup(np.arange(-3, 3), value=10)
    ... # doctest: +NORMALIZE_WHITESPACE
    array([-3, -2, 10, 0, 1, 2])

    """
    result = []
    for i in range(len(x)):
        if x[i] == missing:
            result.append(value)
        else:
            result.append(x[i])
    return np.array(result)

Spoilers below. You'll get much the best results if you have a go yourself before looking at my solutions!

Answer 1

np.sum(x) * np.sum(y)

Answer 2

np.sum(np.searchsorted(np.sort(x), y))

Answer 3

np.where(x == missing, value, x)

The first simple thing you can do is use broadcasting to turn three calculations into one:

xg[p]=(i * xv[0] + j * xv[1] + k * xv[2] + l * xv[3])/n

The next is to note that the division by n can be moved to the very end:

return xg / n

Then, we can split apart the four multiplies and store the results separately, then combine them at the end. Now we have:

xg = np.empty((nsize,4)) # n.b. zeros not required
p = 0

for i in range ( 0, n + 1 ):
    for j in range ( 0, n + 1 - i ):
        for k in range ( 0, n + 1 - i - j ):
            l = n - i - j - k
            xg[p,0] = i
            xg[p,1] = j
            xg[p,2] = k
            xg[p,3] = l
            p = p + 1

return (xg[:,:,None] * xv).sum(1) / n

The trick of xg[:,:,None] at the bottom is to broadcast (nsize,n) * (n,3) - we expand the (nsize,n) xg to (nsize,n,3) because i,j,k,l do not depend on which column of xv is being multiplied with.

We can skip computing l in the loop and instead do it all at once right before the return:

xg[:,3] = n - xg[:,0:3].sum(1)

Now all you need to do is figure out how to create i,j,k in a vectorized way according to p.

As a general note, I find it easiest to work these problems from the "inside out," looking at the code in the innermost loop and pushing as much as possible outside of as many loops as possible. Do this over and over, and eventually there will be no loops.

Maybe you could replace the inner loop and your function with these matrix operations?

for i in range(307200):
    temp = A[:,i,np.newaxis] - B[:]
    C[i,:] = np.linalg.norm(temp, axis=0)

For smaller arrays, I got about 20 times lower running times. Perhaps you gain even more. In any case, plese make sure that you are receiving good results (on a smaller sets).

You need to multiply values from the same date together and add them up; One way to do this is to aggregate the result using a dictionary with date as the key; here is an example using a defaultdict with zero as the default value:

from collections import defaultdict

# aggregate the result into d
d = defaultdict(int)
​
for date, x, y in playground:    
    calc = Calculates(x,y).Calculator()
    d[date] += calc
​
# print the dictionary
for date, calc in d.items():
    print(date, calc)
# 2017-11-11 1859
# 2017-11-12 1359

There seem to be a load of unnecessary variables in your code, like count, X, and i. Removing them and using a standard for loop, you get:

X=[0,1,1,1,0]
a=0
for val in X:
    print (a)
    HWPQ=np.matrix([
        [1, 0, 0, 0], 
        [0, math.cos(4*math.radians(a)), math.sin(4*math.radians(a)), 0], 
        [0, math.sin(4 * math.radians(a)), - math.cos(4 * math.radians(a)), 0],
        [0, 0, 0, -1]])
    result=HWPQ*val
    print (result)
    a=15+a
    print ("\n")

which creates the output that you want.

0
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]


15
[[ 1.         0.         0.         0.       ]
 [ 0.         0.5        0.8660254  0.       ]
 [ 0.         0.8660254 -0.5        0.       ]
 [ 0.         0.         0.        -1.       ]]


30
[[ 1.         0.         0.         0.       ]
 [ 0.        -0.5        0.8660254  0.       ]
 [ 0.         0.8660254  0.5        0.       ]
 [ 0.         0.         0.        -1.       ]]


45
[[  1.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]
 [  0.00000000e+00  -1.00000000e+00   1.22464680e-16   0.00000000e+00]
 [  0.00000000e+00   1.22464680e-16   1.00000000e+00   0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00  -1.00000000e+00]]


60
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]