You are performing two completely different operations here so you cannot directly compare: multiplying a list by 2 will create a new list where the list is concatenated to itself (so the length of the output list is twice that of the input list), whereas multiplying a numpy array by 2 will create a new array of the same length as the original array, but in which each element has been multiplied by 2.
Nonetheless, if you had attempted to perform a list operation which actually corresponds to the numpy case (element-by-element multiplication), for example:
my_list2 = [n * 2 for n in my_listy]
you would also have found that the numpy example was quicker. This is because the required looping in numpy is performed in a shared library consisting of compiled C code, rather than using an explicit loop in Python (for loop or list comprehension).
$ python -mtimeit -s 'import numpy as np; my_array = np.array(range(1000000))' 'my_array2 = my_array * 2'
1000 loops, best of 3: 1.45 msec per loop
$ python -mtimeit -s 'my_listy = list(range(1000000))' 'my_list2 = [n*2 for n in my_listy]'
10 loops, best of 3: 50.8 msec per loop
Why is NumPy Faster Than Lists?
w3schools says
NumPy arrays are stored at one continuous place in memory unlike lists, so processes can access and manipulate them very efficiently. This behavior is called locality of reference in computer science. This is the main reason why NumPy is faster than lists.
That line seems to suggest List elements are not stored contiguously, which contrasts with my understanding that array data structures in all languages are designed to occupy a contiguous block of memory, as described in this Python book.
You are performing two completely different operations here so you cannot directly compare: multiplying a list by 2 will create a new list where the list is concatenated to itself (so the length of the output list is twice that of the input list), whereas multiplying a numpy array by 2 will create a new array of the same length as the original array, but in which each element has been multiplied by 2.
Nonetheless, if you had attempted to perform a list operation which actually corresponds to the numpy case (element-by-element multiplication), for example:
my_list2 = [n * 2 for n in my_listy]
you would also have found that the numpy example was quicker. This is because the required looping in numpy is performed in a shared library consisting of compiled C code, rather than using an explicit loop in Python (for loop or list comprehension).
$ python -mtimeit -s 'import numpy as np; my_array = np.array(range(1000000))' 'my_array2 = my_array * 2'
1000 loops, best of 3: 1.45 msec per loop
$ python -mtimeit -s 'my_listy = list(range(1000000))' 'my_list2 = [n*2 for n in my_listy]'
10 loops, best of 3: 50.8 msec per loop
The following are the main reasons behind the fast speed of Numpy.
-Numpy array is a collection of similar data-types that are densely packed in memory. A Python list can have different data-types, which puts lots of extra constraints while doing computation on it.
-Numpy is able to divide a task into multiple subtasks and process them parallelly.
-Numpy functions are implemented in C. Which again makes it faster compared to Python Lists.
Python at first was not made for numeric operations but with time numpy was created for this scope(making python better at numeric operations). Source: towardsdatascience.com
Numpy arrays are densely packed arrays of homogeneous type. Python lists, by contrast, are arrays of pointers to objects, even when all of them are of the same type. So, you get the benefits of locality of reference.
Also, many Numpy operations are implemented in C, avoiding the general cost of loops in Python, pointer indirection and per-element dynamic type checking. The speed boost depends on which operations you're performing, but a few orders of magnitude isn't uncommon in number crunching programs.
numpy arrays are specialized data structures. This means you don't only get the benefits of an efficient in-memory representation, but efficient specialized implementations as well.
E.g. if you are summing up two arrays the addition will be performed with the specialized CPU vector operations, instead of calling the python implementation of int addition in a loop.
NumPy's arrays are more compact than Python lists -- a list of lists as you describe, in Python, would take at least 20 MB or so, while a NumPy 3D array with single-precision floats in the cells would fit in 4 MB. Access in reading and writing items is also faster with NumPy.
Maybe you don't care that much for just a million cells, but you definitely would for a billion cells -- neither approach would fit in a 32-bit architecture, but with 64-bit builds NumPy would get away with 4 GB or so, Python alone would need at least about 12 GB (lots of pointers which double in size) -- a much costlier piece of hardware!
The difference is mostly due to "indirectness" -- a Python list is an array of pointers to Python objects, at least 4 bytes per pointer plus 16 bytes for even the smallest Python object (4 for type pointer, 4 for reference count, 4 for value -- and the memory allocators rounds up to 16). A NumPy array is an array of uniform values -- single-precision numbers takes 4 bytes each, double-precision ones, 8 bytes. Less flexible, but you pay substantially for the flexibility of standard Python lists!
NumPy is not just more efficient; it is also more convenient. You get a lot of vector and matrix operations for free, which sometimes allow one to avoid unnecessary work. And they are also efficiently implemented.
For example, you could read your cube directly from a file into an array:
x = numpy.fromfile(file=open("data"), dtype=float).reshape((100, 100, 100))
Sum along the second dimension:
s = x.sum(axis=1)
Find which cells are above a threshold:
(x > 0.5).nonzero()
Remove every even-indexed slice along the third dimension:
x[:, :, ::2]
Also, many useful libraries work with NumPy arrays. For example, statistical analysis and visualization libraries.
Even if you don't have performance problems, learning NumPy is worth the effort.