I recently came across the benefits of deque over list.
Should I be using deque instead of list always now ? Is there a scenario where I would need list instead ?
Thanks for help.
The article mentions the advantages of deque, but neglects the mention its disadvantages. While deque has fast pop/append from both ends, it has slow item access.
Use deque only if you need insert/remove to be fast from both ends, and don't care about read speeds. List has constant time append/pop from one end and also constant time access anywhere in the list. You should almost always prefer list over deque, which is why list is builtin.
I wrote a bit of test code based largely on the OP's code and tested a few different operations:
-
del a[0]
-
del a[-1]
-
a.pop()
-
a.append()
-
a.insert(0) # not for python 2
The results show that using a list can be a lot slower than a deque in certain cases. In other cases the list operations are faster than the deque operations. The code is here and the results are here. The code also runs under python 2, and the results for that are here.
The results for python 3 show that "del a[0]" on a list is slower for large N, but for small N < 25 "del a[0]" is faster. Doing "del a[-1]" is always faster for the list when N is large (something greater than 100). And doing "a.insert(0)" on a list is always slower, especially for large N where lists slow down a lot:
Doing "a.insert(0)" on a list and deque using python3
-----------------------------------------------------------------
Size 10: list took 0.000024s, deque took 0.000007s
Size 100: list took 0.000045s, deque took 0.000036s
Size 1000: list took 0.000932s, deque took 0.000369s
Size 10000: list took 0.039175s, deque took 0.002529s
Size 100000: list took 3.606581s, deque took 0.024473s
-----------------------------------------------------------------
For me, the takeaway is: just get the algorithm right first using lists or deques as you wish. You can, of course, use one or the other if you know it's always better. Then, if performance is lacking, you must profile your code. Using a list or deque just because you think it's faster smells of premature optimization. Get the code working correctly first, then stress it with large amounts of data to see if it's fast enough.
Should I use a Python deque or list as a stack? - Stack Overflow
deque over lists?
design - Are there any use cases for List when Deques and Arrays are available? - Software Engineering Stack Exchange
Use deque instead of list always ?
The article mentions the advantages of deque, but neglects the mention its disadvantages. While deque has fast pop/append from both ends, it has slow item access.
Use deque only if you need insert/remove to be fast from both ends, and don't care about read speeds. List has constant time append/pop from one end and also constant time access anywhere in the list. You should almost always prefer list over deque, which is why list is builtin.
More on reddit.comCould anyone explain me what I did wrong here
Yes, your timing is dominated by the time to create the list or deque. The time to do the pop is insignificant in comparison.
Instead you should isolate the thing you're trying to test (the pop speed) from the setup time:
In [1]: from collections import deque
In [2]: s = list(range(1000))
In [3]: d = deque(s)
In [4]: s_append, s_pop = s.append, s.pop
In [5]: d_append, d_pop = d.append, d.pop
In [6]: %timeit s_pop(); s_append(None)
10000000 loops, best of 3: 115 ns per loop
In [7]: %timeit d_pop(); d_append(None)
10000000 loops, best of 3: 70.5 ns per loop
That said, the real differences between deques and list in terms of performance are:
Deques have O(1) speed for appendleft() and popleft() while lists have O(n) performance for insert(0, value) and pop(0).
List append performance is hit and miss because it uses realloc() under the hood. As a result, it tends to have over-optimistic timings in simple code (because the realloc doesn't have to move data) and really slow timings in real code (because fragmentation forces realloc to move all the data). In contrast, deque append performance is consistent because it never reallocs and never moves data.
For what it is worth:
Python 3
deque.pop vs list.pop
> python3 -mtimeit -s 'import collections' -s 'items = range(10000000); base = [*items]' -s 'c = collections.deque(base)' 'c.pop()'
5000000 loops, best of 5: 46.5 nsec per loop
> python3 -mtimeit -s 'import collections' -s 'items = range(10000000); base = [*items]' 'base.pop()'
5000000 loops, best of 5: 55.1 nsec per loop
deque.appendleft vs list.insert
> python3 -mtimeit -s 'import collections' -s 'c = collections.deque()' 'c.appendleft(1)'
5000000 loops, best of 5: 52.1 nsec per loop
> python3 -mtimeit -s 'c = []' 'c.insert(0, 1)'
50000 loops, best of 5: 12.1 usec per loop
Python 2
> python -mtimeit -s 'import collections' -s 'c = collections.deque(xrange(1, 100000000))' 'c.pop()'
10000000 loops, best of 3: 0.11 usec per loop
> python -mtimeit -s 'c = range(1, 100000000)' 'c.pop()'
10000000 loops, best of 3: 0.174 usec per loop
> python -mtimeit -s 'import collections' -s 'c = collections.deque()' 'c.appendleft(1)'
10000000 loops, best of 3: 0.116 usec per loop
> python -mtimeit -s 'c = []' 'c.insert(0, 1)'
100000 loops, best of 3: 36.4 usec per loop
As you can see, where it really shines is in appendleft vs insert.
I found this method while searching for more efficient way to use lists: https://docs.python.org/2/library/collections.html#collections.deque
Its o(1) instead of lists o(n), with this comes mine question. There is a situation where lists are preferred over deque?
A List is general and flexible.
Generalized structures are ideal for return values in business, web, and desktop applications. It's not optimal to return a Deque or an Array if your consumers will be converting to List in most cases anyway. It's potentially less efficient. And, it clutters your consumers' codebases with boilerplate conversions and/or utility libraries to deal with your "more efficient" interfaces.
To be clear: Some structures, like Deque's and Array's are certainly more efficient for particular use cases, but a List is sufficiently efficient for most use cases. List's may be over-used. But, in many domains, the performance hit of using a List over a more well-suited structure is negligible — not even worth a second thought.
Compound that with work-effort versus business value: Choosing a data structure that supports the precise interface I need and no more requires mental effort and time. And those are costs to the business. Sometimes there's a corresponding increase in value. But, in many applications there isn't — neither the business nor the customers will see any difference if you swap your List out for a more-optimized Deque when you're piping 50 rows from the database through your model to your view. (Or whatever.)
That said, there are use-cases that are well-served by List's.
The "need" for a List is more obvious in long-lived, stateful applications, where collections can stick around and be mutated over a long period of time. A text editor, for example, shouldn't automatically create a data[MAX_ROWS][MAX_COLS] array of arrays. And, you certainly could create an array of arrays with some default sizes and scaling behaviors and add methods for insertions and deletions. But, that's precisely what the List interface already does for you. (And the existing implementation already well-tested and optimized!)
My "2 cents": Unless your domain is small, very well-defined, or performance-critical, start with general collections and optimize later. The flexibility is immediate valuable in most business and web applications. Whereas the performance edge is negligible.
And don't forget about stateful applications, where a List interface is more often precisely what you'll end up reinventing if you've defaulted to an array or Deque...
Examples of things where you might favor efficiency over flexibility might include things like system code (OS, driver, firmware), a VM, or other types of "platform" code — like a modern web browser's internal DOM implementation or JavaScript execution engine.
Is there a reason to use java.util.List instead of using java.util.Deque?
Yes.
java.util.List has methods to access elements by position. java.util.Deque does not. If you are implementing an algorithm that needs to access elements by position, then Deque is not going to work. The contrapositive is true as well: If your algorithm only needs to look at the items in sequence (i.e., if you can use an Iterator), and if it only needs to mutate the sequence at the two ends (or one end, or not at all), then a Deque will work for you.
The guiding principle is pretty simple: Choose an interface that defines the methods that your application needs to call.
Is java.util.List a well-designed inteface with a clear purpose?
IMO, No. It tries to be too many different things. If I want you to call my function that will randomly access the list elements, then I am not going to allow you to pass in just any List. I am going to make you pass in an ArrayList, or maybe, if we are collaborating on a project where it makes sense to have a lot of custom classes, we will define an interface that defines exactly those operations that my code is going to perform on your list.