Just to answer part of the question: popping from the end (the right end) of a list takes constant time in CPython, but popping from the left end (.pop(0)) takes time proportional to the length of the list: all the elements in the_list[1:] are physically moved one position to the left.
If you need to delete index position 0 frequently, much better to use an instance of collections.deque. Deques support efficient pushing and popping from both ends.
BTW, when I run the program, I get a clean exception:
...
length of pmarbs = 8306108
Traceback (most recent call last):
File "xxx.py", line 22, in <module>
pmarbs.append(pot2)
MemoryError
That happened to be on a 32-bit Windows box. And it doesn't surprise me ;-)
Answer from Tim Peters on Stack OverflowJust to answer part of the question: popping from the end (the right end) of a list takes constant time in CPython, but popping from the left end (.pop(0)) takes time proportional to the length of the list: all the elements in the_list[1:] are physically moved one position to the left.
If you need to delete index position 0 frequently, much better to use an instance of collections.deque. Deques support efficient pushing and popping from both ends.
BTW, when I run the program, I get a clean exception:
...
length of pmarbs = 8306108
Traceback (most recent call last):
File "xxx.py", line 22, in <module>
pmarbs.append(pot2)
MemoryError
That happened to be on a 32-bit Windows box. And it doesn't surprise me ;-)
list.pop(index) is an O(n) operation, because after you remove the value from the list, you have to shift the memory location of every other value in the list over one. Calling pop repeatedly on large lists is great way to waste computing cycles. If you absolutely must remove from the front of a large list over and over use collections.deque, which will give you much faster insertions and deletions to thr front.
len() is O(1) because deletions are O(n), since if you make sure all the values in a list are allocated in memory right next to each other, the total length of a list is just the tail's memory location - the head's memory location. If you don't care about the performance of len() and similar operations, then you can use a linked list to do constant time insertions and deletions - that just makes len() be O(n) and pop() be O(1) (and you get some other funky stuff like O(n) lookups).
Everything I said about pop() goes for insert() also - except for append(), which usually takes O(1).
I recently worked on a problem that required deleting lots of elements from a very large list (around 10,000,000 integers) and my initial dumb implementation just used pop() every time I needed to delete something - that turned out to not work at all, because it took O(n) to do even one cycle of the algorithm, which itself needed to n times.
My solution was to create a set() called ignore in which I kept the indices of all "deleted" elements. I had little helper functions to help me not have to think about skipping these, so my algorithm didn't get too ugly. What eventually did it was doing a single O(n) pass every 10,000 iterations to delete all the elements in ignore and make ignore empty again, that way I got the increased performance from a shrinking list while only having to do one 10,000th of the work for my deletions.
Also, ya, you should get a memory error because you are trying to allocate a list that is definitely much larger than your hard drive - much less your memory.
I was surprised at how slow list.pop() is! And list.remove() is even many times slower
What is the most efficient way to push and pop a list in Python? - Stack Overflow
What is the time complexity of popping elements from list in Python? - Stack Overflow
performance - Python list pop() much slower than list[1:] - Stack Overflow
I know there is a list.clear(), I'm just sharing that I didn't expect that using list.pop() and list.remove() specifically could slow down the program that much.
li = list(range(500000))
Creating a list is quick.
So we are going to test out pop/remove specific values. For the purpose of this "benchmark", we are going to remove all elements from the list:
while (li):
li.pop(0)
It took 74.735 seconds to pop all the elements! It's ridiculously long.
I KNOW it would have been much faster if I even had used li.pop() without the index or maybe used filter function, list comprehension with conditional or whatever
But that's what I'm trying to show, how slow it is to remove certain list items specifically using pop and remove methods.
And li.remove(), which always requires a specified value to remove, is even worse than pop!
for num in li:
li.remove(num)This one took me 303.268 seconds to complete. How crazy it is.
I've been having fun with abstract data structures. Implemented linked lists and a queues running on linked lists.
And for the sake of interest, I decided to compare the performance of the queue based on the linked list and the usual python list. And I was surprised. When my linked list Queue dequeued 500.000 elements in 0.5 seconds, while python list Queue was doing it in 75 seconds.
Don't use a list.
A list can do fast inserts and removals of items only at its end. You'd use pop(-1) and append, and you'd end up with a stack.
Instead, use collections.deque, which is designed for efficient addition and removal at both ends. Working on the "front" of a deque uses the popleft and appendleft methods. Note, "deque" means "double ended queue", and is pronounced "deck".
L = [1, 2, 3]
L.pop() # returns 3, L is now [1, 2]
L.append(4) # returns None, L is now [1, 2, 4]
L.insert(0, 5) # returns None, L is now [5, 1, 2, 4]
L.remove(2) # return None, L is now [5, 1, 4]
del(L[0]) # return None, L is now [1, 4]
L.pop(0) # return 1, L is now [4]
Yes, it is O(1) to pop the last element of a Python list, and O(N) to pop an arbitrary element (since the whole rest of the list has to be shifted).
Here's a great article on how Python lists are stored and manipulated: An Introduction to Python Lists.
Pop() for the last element ought to be O(1) since you only need to return the element referred to by the last element in the array and update the index of the last element. I would expect pop() for an arbitrary element to be O(N) and require on average N/2 operations since you would need to move any elements beyond the element you are removing one position up in the array of pointers.
You have measured wrong. With cPython 2.7 on x64, I get the following results:
$ python -m timeit 'l = list(range(10000))' 'while l: l = l[1:]'
10 loops, best of 3: 365 msec per loop
$ python -m timeit 'l = list(range(10000))' 'while l: l.pop()'
1000 loops, best of 3: 1.82 msec per loop
$ python -m timeit 'import collections' \
'l = collections.deque(list(range(10000)))' 'while l: l.pop()'
1000 loops, best of 3: 1.67 msec per loop
Use generators for perfomance
python -m timeit 'import itertools' 'l=iter(xrange(10000))' 'while next(l, None): l,a = itertools.tee(l)'
1000000 loops, best of 3: 0.986 usec per loop