algorithm - Why does Python take O(n) time to remove the first element from a list? - Stack Overflow
python 2.7 set and list remove time complexity - Stack Overflow
optimization - How does Python remove elements from a list so quickly? - Stack Overflow
What is the time complexity of popping elements from list in Python? - Stack Overflow
I was trying to write a simple application, which is ao supposed to filter a list of words down to a list of words of a certain length. For that I could either remove the words of the wrong length, or create a new list of words with the correct length.
I had a list of around 58000 words, and wanted to filter out all the 6 letter words, which are around 6900.
with open('words.txt') as f:
words = f.readlines()
for i in range(len(words)):
words[i] = words[i].strip()
length = int(input("Desired word length "))
for i in reversed(words):
if len(i) != length:
words.remove(i)This took 22 seconds.
Another way is to just create a new list with words of the correct length. I did this as follows:
with open('words.txt') as f:
words = f.readlines()
for i in range(len(words)):
words[i] = words[i].strip()
length = int(input("Desired word length "))
clw = []
for i in words:
if len(i) == length:
clw.append(i)This only took 0.03 seconds. How can it be that creating a list of 6900 words takes 0.03 seconds, but removing 51100 words takes 22? It's only 7 times as many words, but takes 700 times as long. And is there a better and faster way to quickly remove list elements?
The pointer to where the list really starts must be retained for the purpose of freeing the memory appropriately.
Indeed, remove(0) could be made faster by having a second pointer which is increased in this case. And if an .add(0, x) happens afterwards, this could be made faster by decrementing this "data start timer" as long as it is bigger than the "memory start timer".
But all other operations, i. e. insertions and deletions to other indexes, would still be O(n), so that wouldn't change much.
Just know what your operations will be and thus which data structure to pick.
Python list is actually an array. deque is a real linked list. It is Python's fault for using the wrong term (for which I do not have an explanation). O(n) for insertion and deletion is normal for arrays (as following elements need to be shifted up or down), which is a tradeoff for the O(1) speed for get and set. Linked lists make a similar tradeoff in the opposite direction: O(1) for operations at ends, but O(n) for any access in the middle.
I decided to turn my set of comments into a proper answer.
First off, let's clarify what's happening when you do:
>>> l = [i for i in range(100000000)]
Here three things are happening:
- 100000000
intobjects are being created. Creating an object in CPython requires allocating memory and placing content into that memory, and this takes time. - You are running a loop. This affects performances considerably:
[i for i in range(...)]is much slower thanlist(range(...)). - The large list is being created on the fly.
Reading your question, it seems that you are considering only the last point, ignoring the others. Therefore, your timings are inaccurate: creating a large list does not take 3 seconds, it takes a fraction of those 3 seconds.
How large is this fraction is an interesting question, which is difficult to answer using only Python code, but still we can try. Specifically, I'd try with the following statement:
>>> [None] * 100000000
Here CPython does not have to create a large amount of objects (there's only None), does not have to run loops and can allocate the memory for the list once (because it knows the size in advance).
Timings are self-explanatory:
$ python3 -m timeit "list(range(100000000))"
10 loops, best of 3: 2.26 sec per loop
$ python3 -m timeit "[None] * 100000000"
10 loops, best of 3: 375 msec per loop
Now, back to your question: how about item deletion?
$ python3 -m timeit --setup "l = [None] * 100000000" "del l[0]"
10 loops, best of 3: 89 msec per loop
$ python3 -m timeit --setup "l = [None] * 100000000" "del l[100000000 // 4]"
10 loops, best of 3: 66.5 msec per loop
$ python3 -m timeit --setup "l = [None] * 100000000" "del l[100000000 // 2]"
10 loops, best of 3: 45.3 msec per loop
These numbers tell us something important. Note that 2 × 45.3 ≈ 89. Also 66.5 × 4 / 3 ≈ 89.
These numbers are telling exactly what linear complexity is about. If a function has time complexity kn (which is O(n)), it means that if we double the input, we double time; if we increase the input size by 4/3, the time increases by 4/3.
And this is what's happening here. In CPython, our list of 100000000 items is a contiguous memory area containing pointers to Python objects:
l = |ptr0|ptr1|ptr2|...|ptr99999999|
When we run del l[0] we are moving ptr1 from right to left, overwriting ptr0. The same for other elements:
l = |ptr0|ptr1|ptr2|...|ptr99999999|
^^^^
` item to delete
l = |ptr1|ptr2|...|ptr99999999|
Therefore, when we run del l[0] we have to move 99999998 pointers to the left. This is different from del l[100000000 // 2], which requires moving only half the pointers (the pointers on the first half don't need to be moved). "Moving half the pointers" equals "performing half of the operations" which roughly mean "running in half the time" (this is not always true, but timings say that this is true in this case).
I'm not sure why you think it should take 3 seconds to delete a single element.
Your initial time is for 100000000 individual append operations. Each of those takes a fraction of a second; your delete operation takes a similar amount of time.
In any case, as Bartosz points out, O(n) complexity doesn't mean that all operations take the same length of time, it means that the length of time is proportional to the length of the list.
Yes, it is O(1) to pop the last element of a Python list, and O(N) to pop an arbitrary element (since the whole rest of the list has to be shifted).
Here's a great article on how Python lists are stored and manipulated: An Introduction to Python Lists.
Pop() for the last element ought to be O(1) since you only need to return the element referred to by the last element in the array and update the index of the last element. I would expect pop() for an arbitrary element to be O(N) and require on average N/2 operations since you would need to move any elements beyond the element you are removing one position up in the array of pointers.
As you correctly noticed, the CPython implementation of list.clear is O(n). The code iterates over the elements in order to decrease the reference count of each one, without a way to avoid it. There is no doubt that it is an O(n) operation and, given a large enough list, you can measure the time spent in clear() as function of list size:
import time
for size in 1_000_000, 10_000_000, 100_000_000, 1_000_000_000:
l = [None] * size
t0 = time.time()
l.clear()
t1 = time.time()
print(size, t1 - t0)
The output shows linear complexity; on my system with Python 3.7 it prints the following:
1000000 0.0023756027221679688
10000000 0.02452826499938965
100000000 0.23625731468200684
1000000000 2.31496524810791
The time per element is of course tiny because the loop is coded in C and each iteration does very little work. But, as the above measurement shows, even a miniscule per-element factor eventually adds up. Small per-element constant is not the reason to ignore the cost of an operation, or the same would apply to the loop that shifts the list elements in l.insert(0, ...), which is also very efficient - and yet few would claim insertion at the beginning to be O(1). (And clear potentially does more work because a decref will run an arbitrary chain of destructors for an object whose reference count actually reaches zero.)
On a philosophical level, one could argue that costs of memory management should be ignored when assessing complexity because otherwise it would be impossible to analyze anything with certainty, as any operation could trigger a GC. This argument has merit; GC does come occasionally and unpredictably, and its cost can be considered amortized across all allocations. In a similar vein complexity analysis tends to ignore the complexity of malloc because the parameters it depends on (like memory fragmentation) are typically not directly related to allocation size or even to the number of already allocated blocks. However, in case of list.clear there is only one allocated block, no GC is triggered, and the code is still visiting each and every list element. Even with the assumption of O(1) malloc and amortized O(1) GC, list.clear still takes the time proportional to the number of elements in the list.
The article linked from the question is about Python the language and doesn't mention a particular implementation. Python implementations that don't use reference counting, such as Jython or PyPy, are likely to have true O(1) list.clear, and for them the claim from the article would be entirely correct. So, when explaining the Python list on a conceptual level, it is not wrong to say that clearing the list is O(1) - after all, all the object references are in a contiguous array, and you free it only once. This is the point your blog post probably should make, and that is what the linked article is trying to say. Taking the cost of reference counting into account too early might confuse your readers and give them completely wrong ideas about Python's lists (e.g. they could imagine that they are implemented as linked lists).
Finally, at some point one must accept that memory management strategy does change complexity of some operations. For example, destroying a linked list in C++ is O(n) from the perspective of the caller; discarding it in Java or Go would be O(1). And not in the trivial sense of a garbage-collected language just postponing the same work for later - it is quite possible that a moving collector will only traverse reachable objects and will indeed never visit the elements of the discarded linked list. Reference counting makes discarding large containers algorithmically similar to manual collection, and GC can remove that. While CPython's list.clear has to touch every element to avoid a memory leak, it is quite possible that PyPy's garbage collector never needs to do anything of the sort, and thus has a true O(1) list.clear.
It's O(1) neglecting memory management. It's not quite right to say it's O(N) accounting for memory management, because accounting for memory management is complicated.
Most of the time, for most purposes, we treat the costs of memory management separately from the costs of the operations that triggered it. Otherwise, just about everything you could possibly do becomes O(who even knows), because almost any operation could trigger a garbage collection pass or an expensive destructor or something. Heck, even in languages like C with "manual" memory management, there's no guarantee that any particular malloc or free call will be fast.
There's an argument to be made that refcounting operations should be treated differently. After all, list.clear explicitly performs a number of Py_XDECREF operations equal to the list's length, and even if no objects are deallocated or finalized as a result, the refcounting itself will necessarily take time proportional to the length of the list.
If you count the Py_XDECREF operations list.clear performs explicitly, but ignore any destructors or other code that might be triggered by the refcounting operations, and you assume PyMem_FREE is constant time, then list.clear is O(N), where N is the original length of the list. If you discount all memory management overhead, including the explicit Py_XDECREF operations, list.clear is O(1). If you count all memory management costs, then the runtime of list.clear cannot be asymptotically bounded by any function of the list's length.