More "modern python" way of doing things.

[[ randint(0,4) for x in range(0,4)] for y in range(0,4)]

Its simply a pair of nested list comprehensions.

Are you necessarily looking to use for-loops? I am asking because this problem can be solved in simpler and more efficient ways such as:

import numpy as np

a_values = np.random.rand(20,402) #Store random values in a_values with a shape of 20 rows and 402 columns
avg_columns = a_values.mean(axis=0) #Calculate the mean of each column 

print(avg_columns)

Documentation for generating random values: numpy.random.rand

EDIT:

I assumed that xa is randomized in each iteration (make sure to replace it with the xa that you are generating). You can initialize an empty a_values array, and add the generated xa array to it in each iteration using np.vstack. For the first iteration, when a_value is empty, I made it equal to xa (if a_values.size else xa).

import numpy as np

a_values = np.array([])

for i in range(20):
  xa = np.random.rand(1,402)
  a_values = np.vstack([a_values, xa]) if a_values.size else xa

avg_columns = a_values.mean(axis=0) #Average of each column

Documentation for vstack: numpy.vstack

There are fancier ways, but this is the most straightforward:

>>> rows = 3
>>> columns = 7
>>> n = 1
>>> matrix = []
>>> for _ in range(rows):
...     sub = []
...     for _ in range(columns):
...         sub.append(n)
...         n += 1
...     matrix.append(sub)
...
>>> matrix
[[1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, 14], [15, 16, 17, 18, 19, 20, 21]]

And for good measure, a fancy way:

>>> import itertools
>>> counter = itertools.count(1)
>>> rows = 3
>>> columns = 7
>>> matrix = [[n for n, _ in zip(counter, range(columns))] for _ in range(rows)]
>>> matrix
[[1, 2, 3, 4, 5, 6, 7], [9, 10, 11, 12, 13, 14, 15], [17, 18, 19, 20, 21, 22, 23]]
>>>

First of all don't name your array dict, it is confusing as well as hides the built-in type dict.

The problem here is that you're doing everything in quadratic time, so convert your arrays dict and id to a dictionary first where each word or id point to its index.

matrix = sparse.lil_matrix((len(ids), len(dict)))
dict_from_dict = {word: ind for ind, word in enumerate(dict)}
dict_from_id = {id: ind for ind, id in enumerate(id)}
for i in words:
    matrix[dict_from_id[i['id']], dict_from_dict[i['word']] += 1.0

The problem here is in line

board[x:y] = player;

the [x:y] notation is for assigning ranges while what you really need is probably

board[x][y] = player

Normally you would get an error because assigning single value to a range will throw TypeError. In this case you are assigning string which python treats as a list of characters let's say.

I know that you can create shared numpy arrays that can be changed from different threads (assuming that the changed areas don't overlap). Here is the sketch of the code that you can use to do that (I saw the original idea somewhere on stackoverflow, edit: here it is https://stackoverflow.com/a/5550156/1269140 )

import multiprocessing as mp ,numpy as np, ctypes

def shared_zeros(n1, n2):
    # create a 2D numpy array which can be then changed in different threads
    shared_array_base = mp.Array(ctypes.c_double, n1 * n2)
    shared_array = np.ctypeslib.as_array(shared_array_base.get_obj())
    shared_array = shared_array.reshape(n1, n2)
    return shared_array

class singleton:
    arr = None

def dosomething(i):
    # do something with singleton.arr
    singleton.arr[i,:] = i
    return i

def main():
    singleton.arr=shared_zeros(1000,1000)
    pool = mp.Pool(16)
    pool.map(dosomething, range(1000))

if __name__=='__main__':
    main()

I guess the easiest way to do this is to reshape an arange from 0 to 100:

>>> np.arange(100).reshape(10, -1)
array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
       [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
       [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
       [50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
       [60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
       [70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
       [80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
       [90, 91, 92, 93, 94, 95, 96, 97, 98, 99]])

Here the .reshape(..) call will thus transform the matrix such that it is a 2D-array, with 10 "rows" and a number of columns such that the total amount of cells is 100.

In case you do not want to construct a 2D-array, but a Python list of 1D arrays, we can use list comprehension:

[np.arange(i, i+10) for i in range(0, 100, 10)]

You can calculate the sizes ahead in essentially constant time. Just do that, and use numpy.fromiter:

In [1]: import math, from itertools import permutations, chain

In [2]: def n_chose_k(n, k, fac=math.factorial):
    ...:     return fac(n)/fac(n-k)
    ...:

In [3]: def permutations_to_array(r, k):
    ...:     n = len(r)
    ...:     size = int(n_chose_k(n, k))
    ...:     it = permutations(r, k)
    ...:     arr = np.fromiter(chain.from_iterable(it),
    ...:                       count=size,  dtype=int)
    ...:     arr.size = size//k, k
    ...:     return arr
    ...:

In [4]: arr = permutations_to_array(range(1,20), 7)

In [5]: arr.shape
Out[5]: (36279360, 7)

In [6]: arr[0:5]
Out[6]:
array([[ 1,  2,  3,  4,  5,  6,  7],
       [ 1,  2,  3,  4,  5,  6,  8],
       [ 1,  2,  3,  4,  5,  6,  9],
       [ 1,  2,  3,  4,  5,  6, 10],
       [ 1,  2,  3,  4,  5,  6, 11]])

This will work as long as r is limited to sequences with a len.

Edited to add an implementation I cooked up for a generator of batchsize*k chunks, with a trim option!

import math
from itertools import repeat, chain

import numpy as np

def n_chose_k(n, k, fac=math.factorial):
    return fac(n)/fac(n-k)

def permutations_in_batches(r, k, batchsize=None, fill=0, dtype=int, trim=False):
    n = len(r)
    size = int(n_chose_k(n, k))
    if batchsize is None or batchsize > size:
        batchsize = size
    perms = chain.from_iterable(permutations(r, k))
    count = batchsize*k
    remaining = size - count
    while remaining > 0:
        current = np.fromiter(perms, count=count, dtype=dtype)
        current.shape = batchsize, k
        yield current
        remaining -= count
    if remaining: # remaining is negative
        remaining = -remaining
        if not trim:
            padding = repeat(fill, remaining)
            finalcount = count
            finalshape = batchsize, k
        else:
            q = remaining//k # always divisible q%k==0
            finalcount = q*k
            padding = repeat(fill, remaining)
            finalshape = q, k
        current =  np.fromiter(chain(perms, padding), count=finalcount, dtype=dtype)
        current.shape = finalshape
    else: # remaining is 0
        current = np.fromiter(perms, count=batchsize, dtype=dtype)
        current.shape = batchsize, k
    yield current

As I explained in my answer to your previous question, you really need to vectorize arbitrary_function.

You can do this by just calling np.vectorize on the function, something like this:

Z = np.vectorize(arbitrary_function)(np.arange(3), np.arange(5).reshape(5, 1))

But that will only give you a small speedup. In your case, since arbitrary_function is doing a huge amount of work (including opening and parsing an Excel spreadsheet), it's unlikely to make enough difference to even notice, much less to solve your performance problem.

The whole point of using NumPy for speedups is to find the slow part of the code that operates on one value at a time, and replace it with something that operates on the whole array (or at least a whole row or column) at once. You can't do that by looking at the very outside loop, you need to look at the very inside loop. In other words, at arbitrary_function.

In your case, what you probably want to do is read the Excel spreadsheet into a global array, structured in such a way that each step in your process can be written as an array-wide operation on that array. Whether that means multiplying by a slice of the array, indexing the array using your input values as indices, or something completely different, it has to be something NumPy can do for you in C, or NumPy isn't going to help you.

If you can't figure out how to do that, you may want to consider not using NumPy, and instead compiling your inner loop with Cython, or running your code under PyPy. You'll still almost certainly need to move the "open and parse a whole Excel spreadsheet" outside of the inner loop, but at least you won't have to figure out how to rethink your problem in terms of vectorized operations, so it may be easier for you.

First you have to install numpy using

$ pip install numpy

Then the following should work

import numpy as np    
n = 100
matrix = np.zeros((n,2)) # Pre-allocate matrix
for i in range(1,n):
    matrix[i,:] = [3*i, i**2]

A faster alternative:

col1 = np.arange(3,3*n,3)
col2 = np.arange(1,n)
matrix = np.hstack((col1.reshape(n-1,1), col2.reshape(n-1,1)))

Even faster, as Divakar suggested

I = np.arange(n)
matrix = np.column_stack((3*I, I**2))

The four sums you want can be calculated efficiently like this:

import numpy as np
arr = np.array(data)

w = arr[0::2,0] # array([ 1,  3,  5, 13])
x = arr[0::2,1] # array([ 7,  9, 11, 14])
y = arr[1::2,0] # array([ 2,  4,  6, 15])
z = arr[1::2,1] # array([ 8, 10, 12, 16])

B = [[w.sum(), x.sum()], [y.sum(), z.sum()]]

arr[1::2,0] means "Starting from row 1, take every second row, then take column 0."

import numpy as np
max_array = 4
ARRAY = np.arange(max_array*2).reshape((max_array,2))

You have created a 2-d array

>>> ARRAY
array([[0, 1],
       [2, 3],
       [4, 5],
       [6, 7]])
>>> 

ARRAY[i,i] indexes a single element in the array

>>> i = 0
>>> ARRAY[i,i]
0
>>> ARRAY[i,i] = 222
>>> ARRAY
array([[222,   1],
       [  2,   3],
       [  4,   5],
       [  6,   7]])

If you want to assign values to a row:

>>> ARRAY[0] = 99, 99
>>> ARRAY
array([[99, 99],
       [ 2,  3],
       [ 4,  5],
       [ 6,  7]])

Or

>>> ARRAY[2,:] = 66, 66
>>> ARRAY
array([[99, 99],
       [ 2,  3],
       [66, 66],
       [ 6,  7]])

The second value in the subscript indexes a column

>>> ARRAY[:, 1]
array([99,  3, 66,  7])

>>> ARRAY[:, 1] = 0
>>> ARRAY
array([[99,  0],
       [ 2,  0],
       [66,  0],
       [ 6,  0]])
>>>

ARRAY[i,i] = data1, data2 : tries to assign two things to a single element in the array - that is why you get an error.

Numpy docs
Indexing, Slicing, Iterating
Indexing