The issue is as you guessed - there is an integer overflow. If you take a look at sum(sum(test_big)), you will notice that the values are wrong there.

The part that is wrong is that integer overflow can occur in your sum() functions which are taking the partial sums.

What I would suggest is making a sum of this array using np.sum() as it does give an appropriate sum despite of data type.

Use count_nonzero to count non-zero (e.g. not False) values:

>>> np.size(a) - np.count_nonzero(a)
2

Because x < 6 is not equal to x, but instead it is comparing each element with 6.

import numpy as np
x = np.array([[5,0,3,3], [7,9,3,5],[2,4,7,6]])
print(x < 6)

Output:

array([[ True,  True,  True,  True],
       [False, False,  True,  True],
       [ True,  True, False, False]])

And np.count_nonzero() is equivalent of counting non False (or True) elements in the output which is:

np.count_nonzero(x < 6)
# 8
np.sum(x < 6)
#8

And for the last part, why they don't add up to 11 is because the last element is 6. So:

np.count_nonzero(x < 6) + np.count_nonzero(x > 6) + np.count_nonzero(x == 6) 
# 12

np.count_nonzero(~np.isnan(data))

~ inverts the boolean matrix returned from np.isnan.

np.count_nonzero counts values that is not 0\false. .sum should give the same result. But maybe more clearly to use count_nonzero

Testing speed:

In [23]: data = np.random.random((10000,10000))

In [24]: data[[np.random.random_integers(0,10000, 100)],:][:, [np.random.random_integers(0,99, 100)]] = np.nan

In [25]: %timeit data.size - np.count_nonzero(np.isnan(data))
1 loops, best of 3: 309 ms per loop

In [26]: %timeit np.count_nonzero(~np.isnan(data))
1 loops, best of 3: 345 ms per loop

In [27]: %timeit data.size - np.isnan(data).sum()
1 loops, best of 3: 339 ms per loop

data.size - np.count_nonzero(np.isnan(data)) seems to barely be the fastest here. other data might give different relative speed results.

Assuming you mean total number of nonzero elements (and not total number of nonzero rows):

In [12]: a = np.random.randint(0, 3, size=(100,100))

In [13]: timeit len(a.nonzero()[0])
1000 loops, best of 3: 306 us per loop

In [14]: timeit (a != 0).sum()
10000 loops, best of 3: 46 us per loop

or even better:

In [22]: timeit np.count_nonzero(a)
10000 loops, best of 3: 39 us per loop

This last one, count_nonzero, seems to behave well when the array is small, too, whereas the sum trick not so much:

In [33]: a = np.random.randint(0, 3, size=(10,10))

In [34]: timeit len(a.nonzero()[0])
100000 loops, best of 3: 6.18 us per loop

In [35]: timeit (a != 0).sum()
100000 loops, best of 3: 13.5 us per loop

In [36]: timeit np.count_nonzero(a)
1000000 loops, best of 3: 686 ns per loop

You may also consider, well, counting the nonzero values:

import numba as nb

@nb.njit()
def count_loop(a):
    s = 0
    for i in a:
        if i != 0:
            s += 1
    return s

I know it seems wrong, but bear with me:

import numpy as np
import numba as nb

@nb.njit()
def count_loop(a):
    s = 0
    for i in a:
        if i != 0:
            s += 1
    return s

@nb.njit()
def count_len_nonzero(a):
    return len(np.nonzero(a)[0])

@nb.njit()
def count_sum_neq_zero(a):
    return (a != 0).sum()

np.random.seed(100)
a = np.random.randint(0, 3, 1000000000, dtype=np.uint8)
c = np.count_nonzero(a)
assert count_len_nonzero(a) == c
assert count_sum_neq_zero(a) == c
assert count_loop(a) == c

%timeit count_len_nonzero(a)
# 5.94 s ± 141 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit count_sum_neq_zero(a)
# 848 ms ± 80.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit count_loop(a)
# 189 ms ± 4.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

It is in fact faster than np.count_nonzero, which can get quite slow for some reason:

%timeit np.count_nonzero(a)
# 4.36 s ± 69.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

(EDITED: I somewhat overlooked that you were using NumPy arrays)

To replace:

sum(map(lambda base1 : base1 == x, base1)) 

or:

count1 = sum([base1 == x for base1 in base1])

the best approach depends if your input is a list or a NumPy array.

• if you have a list, you could use the list.count() method:

base1.count(x)

• if you have a NumPy array, as this seems to be the case, you could use np.count_nonzero() for NumPy arrays:

import numpy as np


np.count_nonzero(base1 == x)

However, this will create a potentially large temporary object. This can be solved by creating your own function and accelerate it with Cython (not shown) or, even better, with Numba, as shown below:

import numba as nb


@nb.jit
def nb_count_equal(arr, value):
    result = 0
    for x in arr:
        if x == value:
            result += 1
    return result

which would also be faster than np.count_nonzero() in this case.

Testing some of these approaches on toy data shows that they give the same result:

np.random.seed(0)  # to ensure reproducible results

arr = np.random.randint(0, 20, 1000)
y = 10

print(sum(map(lambda x: x == y, arr)))
# 41
print(sum([x == y for x in arr]))
# 41
print(np.count_nonzero(arr == y))
# 41
print(nb_count_equal(arr, y))
# 41

with the following timing:

arr = np.random.randint(0, 20, 1000000)
y = 10

%timeit sum(map(lambda x: x == y, arr))
# 1 loop, best of 3: 2.54 s per loop
%timeit sum([x == y for x in arr])
# 1 loop, best of 3: 2.43 s per loop
%timeit np.count_nonzero(arr == y)
# 1000 loops, best of 3: 574 µs per loop
%timeit nb_count_equal(arr, y)
# 1000 loops, best of 3: 224 µs per loop

Note, the previous suggestion of removing the square brackets to avoid creating a temporary list is slower than just having a generator because of the way sum() is implemented, but it would definitely have the advantage of avoiding to create unnecessary temporary lists.

Finally, if you are doing this counting multiple times, it may be more beneficial to do this in a single go with np.unique().

import numpy as np

a = np.array([[1, 0, 1],
              [2, 3, 4],
              [0, 0, 7]])

columns = (a != 0).sum(0)
rows    = (a != 0).sum(1)

The variable (a != 0) is an array of the same shape as original a and it contains True for all non-zero elements.

The .sum(x) function sums the elements over the axis x. Sum of True/False elements is the number of True elements.

The variables columns and rows contain the number of non-zero (element != 0) values in each column/row of your original array:

columns = np.array([2, 1, 3])
rows    = np.array([2, 3, 1])

EDIT: The whole code could look like this (with a few simplifications in your original code):

ANOVAInputMatrixValuesArray = zeros([len(TestIDs), 9], float)
for j, TestID in enumerate(TestIDs):
    ReadOrWrite = 'Read'
    fileName = inputFileName
    directory = GetCurrentDirectory(arguments that return correct directory)
    # use directory or filename to get the CSV file?
    with open(directory, 'r') as csvfile:
        ANOVAInputMatrixValuesArray[j,:] = loadtxt(csvfile, comments='TestId', delimiter=';', usecols=(2,))[:9]

nonZeroCols = (ANOVAInputMatrixValuesArray != 0).sum(0)
nonZeroRows = (ANOVAInputMatrixValuesArray != 0).sum(1)

EDIT 2:

To get the mean value of all columns/rows, use the following:

colMean = a.sum(0) / (a != 0).sum(0)
rowMean = a.sum(1) / (a != 0).sum(1)

What do you want to do if there are no non-zero elements in a column/row? Then we can adapt the code to solve such a problem.