I think the key idea you're missing to implement your PriorityQueue class is that each PriorityQueue instance should have a Heap instance as an attribute. Set it up in __init__:
class PriorityQueue(object):
def __init__(self)
self.heap = Heap()
When a user makes a call to a PriorityQueue method, that method will mostly just make a call to a method of self.heap, with just a little extra work modifying the arguments or the return value. The items to insert into the heap should probably be (priority, value) tuples, since they will compare appropriately (higher priority items comparing higher).
Note that if you compare code written for heapq with your Heap, you'll need to modify the logic for the indexes and priorities, since heapq implements a zero-indexed min-heap, and your code implements a one-indexed max-heap.
Queue.PriorityQueue is a thread-safe class, while the heapq module makes no thread-safety guarantees. From the Queue module documentation:
The
Queuemodule implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. TheQueueclass in this module implements all the required locking semantics. It depends on the availability of thread support in Python; see thethreadingmodule.
The heapq module offers no locking, and operates on standard list objects, which are not meant to be thread-safe.
In fact, the PriorityQueue implementation uses heapq under the hood to do all prioritisation work, with the base Queue class providing the locking to make this thread-safe. See the source code for details.
This makes the heapq module faster; there is no locking overhead. In addition, you are free to use the various heapq functions in different, novel ways, the PriorityQueue only offers the straight-up queueing functionality.
queue.PriorityQueue is a partial wrapper around the heapq class.
In other words, a queue.PriorityQueue is actually a heapq, placed in the queue module with a couple of renamed methods to make the heapq easier to use, much like a regular queue.
In heapq, you use use the method heappush() to add a new item and the method heappop() to remove one. That is not very queue-like, so queue.PriorityQueue let you use the usual queue methods such as put and get to do the same thing.
There are some features of heapq that are not carried over into queue.PriorityQueue, such as heappushpop() and heapreplace(), but you are less likely to use those. If you need them (and I do in my current project), perhaps you should use heapq rather than queue.PriorityQueue.
Also, since heapq is specialized for its purpose, it is not thread safe (as noted in another answer here.)
Videos
I know python has heapq and queue.priorityqueue but honestly, both of them are really cumbersome compared to Java's priorityqueue. I find it tedious to have to insert a tuple, with the first element in the tuple defining the priority. Also, it makes it hard to write more complex comparisons. Is there a way we can pass in a comparator to the Priorityqueue? I know it's possible to define classes with their own comparator method, but again, this is really tedious and I'm looking for something as close as possible to Java's PQ.
It depends on what your definition of priority is, but you're going to have to iterate over your collection yourself, looking for the place to insert the next node:
node = self.head
while node.next and node.next.priority > priority:
node = node.next
if node.next is None:
node.next = qNode(value=value, next=None, priority=priority)
self.foot = node.next
else:
new_node = qNode(value=value, next=node.next, priority=priority)
node.next = new_node
You'll have to add a priority to your qNode, of course, and you may have to adjust exactly where you're going to insert, but this should get you most of the way there.
Here is the implementation of priority queue in python using linked list:-
# Python code to implement Priority Queue using Linked List
# Node class
class Node:
def __init__(self, item, priority):
self.item = item
self.next = None
self.priority = priority
class PriorityQueue:
def __init__(self):
self.front = self.rear = None
# Returns a boolean value indicating whether the queue is empty
def isEmpty(self):
return self.front == None
# Adds the given item to the queue by inserting it in the proper
# position based on the given priority. The new node is appended to
# the end of the linked list
def enqueue(self, item, priority):
newNode = Node(item, priority)
if not self.rear:
self.front = self.rear = newNode
return
if self.front.priority < newNode.priority:
newNode.next = self.front
self.front = newNode
return
previous = None
current = self.front
while(current and newNode.priority < current.priority):
previous = current
current = current.next
if current:
previous.next = newNode
newNode.next = current
else:
self.rear.next = newNode
self.rear = newNode
# Removes and returns the next item from the queue, which is the
# item with the highest priority. If two or more items have the
# same priority, those items are removed in FIFO order. An item
# cannot be dequeued from an empty queue.
def dequeue(self):
if self.isEmpty():
print('Queue is empty')
return
temp = self.front
self.front = self.front.next
if self.front == None:
self.rear = None
return temp.item
I created a Python heap implementation that supports:
-
Removing any item (not just the root via pop)
-
Tracking the frequency of items so that duplicates are handled efficiently
Source: https://github.com/Ite-O/python-indexed-heap
PyPI: https://pypi.org/project/indexedheap/
What My Project Does
indexedheap is a Python package that provides standard heap operations, insert (push), pop, and peek, along with additional features:
-
Remove any arbitrary item efficiently.
-
Track frequencies of items to handle duplicates.
-
Insert or remove multiple occurrences in a single operation.
-
Iterate over heap contents in sorted order without modifying the heap.
It is designed for scenarios requiring dynamic priority queues, where an item’s priority may change over time Common in task schedulers, caching systems or pathfinding algorithms.
Target Audience
-
Developers needing dynamic priority queues where priorities can increase or decrease.
-
Users who want duplicate-aware heaps for frequency tracking.
-
Engineers implementing task schedulers, caches, simulations or pathfinding algorithms in Python.
Comparison
Python’s built-in heapq vs indexedheap
| Operation | Description |
heapq
|
indexedheap
|
|---|---|---|---|
heappush(heap, item) / insert(value)
| Add an item/value to the heap |
✅ O(log N)
|
✅ O(log N) / (O(1) if item already exists and count is incremented)
|
heappop(heap) / pop()
| Remove and return the root item/value |
✅ O(log N)
|
✅ O(log N)
|
heap[0] / peek()
| Return root item/value without removing it |
✅ Manual (heap[0])
|
✅ O(1)
|
remove(value)
| Remove any arbitrary value | ❌ Not supported |
✅ O(log(N)) for last occurence in heap, O(1) if only decrementing frequency
|
heappushpop(heap, item)
| Push then pop in a single operation |
✅ O(log N)
|
❌ Not directly supported (use insert() + pop())
|
heapreplace(heap, item)
| Pop then push in a single operation |
✅ O(log N)
|
❌ Not directly supported (use pop() + insert())
|
count(value)
| Get frequency of a specific value | ❌ Not supported |
✅ O(1)
|
item in heap / value in heap
| Membership check |
⚠️ O(N) (linear scan)
|
✅ O(1)
|
len(heap)
| Number of elements |
✅ O(1)
|
✅ O(1)
|
to_sorted_list()
| Return sorted elements without modifying heap |
✅ Requires creating a sorted copy of the heap O(N log N)
|
✅ O(N log N)
|
iter(heap)
| Iterate in sorted order |
✅ Requires creating a sorted copy of the heap and iterating over the copy O(N log N))
|
✅ O(N log N)
|
heapify(list) / MinHeap(list), MaxHeap(list)
| Convert list to valid heap |
✅ O(N)
|
✅ O(N)
|
heap1 == heap2
| Structural equality check |
✅ O(N)
|
✅ O(N)
|
Frequency tracking
| Track frequency of items rather than store duplicates | ❌ Not supported | ✅ Yes |
Multi-inserts/removals
| Insert/ remove multiples of an item in a single operation | ❌ Not supported | ✅ Yes (see insert/ remove rows for time complexity) |
Installation
pip install indexedheap
Feedback
If there is demand, I am considering adding support for heappushpop and heapreplace operations, similar to those in Python's heapq module.
Open to any feedback!
Updates
-
Updated terminology in the comparison table to show both "value" and "item" in some rows. Since the terminology used in my package for the inserted object is "value", whereas the terminology used in heapq is "item".
There is no such thing as a "most efficient priority queue implementation" in any language.
A priority queue is all about trade-offs. See http://en.wikipedia.org/wiki/Priority_queue
You should choose one of these two, based on how you plan to use it:
O(log(N))insertion time andO(1)(findMin+deleteMin)* time, orO(1)insertion time andO(log(N))(findMin+deleteMin)* time
(* sidenote: the findMin time of most queues is almost always O(1), so here I mostly mean the deleteMin time can either be O(1) quick if the insertion time is O(log(N)) slow, or the deleteMin time must be O(log(N)) slow if the insertion time is O(1) fast. One should note that both may also be unnecessarily slow like with binary-tree based priority queues.)
In the latter case, you can choose to implement a priority queue with a Fibonacci heap: http://en.wikipedia.org/wiki/Heap_(data_structure)#Comparison_of_theoretic_bounds_for_variants (as you can see, heapq which is basically a binary tree, must necessarily have O(log(N)) for both insertion and findMin+deleteMin)
If you are dealing with data with special properties (such as bounded data), then you can achieve O(1) insertion and O(1) findMin+deleteMin time. You can only do this with certain kinds of data because otherwise you could abuse your priority queue to violate the O(N log(N)) bound on sorting. vEB trees kind of fall under a similar category, since you have a maximum set size (O(log(log(M)) is not referring to the number of elements, but the maximum number of elements) and thus you cannot circumvent the theoretical O(N log(N)) general-purpose comparison-sorting bound.
To implement any queue in any language, all you need is to define the insert(value) and extractMin() -> value operations. This generally just involves a minimal wrapping of the underlying heap; see http://en.wikipedia.org/wiki/Fibonacci_heap to implement your own, or use an off-the-shelf library of a similar heap like a Pairing Heap (a Google search revealed http://svn.python.org/projects/sandbox/trunk/collections/pairing_heap.py )
If you only care about which of the two you referenced are more efficient (the heapq-based code from http://docs.python.org/library/heapq.html#priority-queue-implementation-notes which you included above, versus Queue.PriorityQueue), then:
There doesn't seem to be any easily-findable discussion on the web as to what Queue.PriorityQueue is actually doing; you would have to source dive into the code, which is linked to from the help documentation: http://hg.python.org/cpython/file/2.7/Lib/Queue.py
Copy 224 def _put(self, item, heappush=heapq.heappush):
225 heappush(self.queue, item)
226
227 def _get(self, heappop=heapq.heappop):
228 return heappop(self.queue)
As we can see, Queue.PriorityQueue is also using heapq as an underlying mechanism. Therefore they are equally bad (asymptotically speaking). Queue.PriorityQueue may allow for parallel queries, so I would wager that it might have a very slightly constant-factor more of overhead. But because you know the underlying implementation (and asymptotic behavior) must be the same, the simplest way would simply be to run them on the same large dataset.
(Do note that Queue.PriorityQueue does not seem to have a way to remove entries, while heapq does. However this is a double-edged sword: Good priority queue implementations might possibly allow you to delete elements in O(1) or O(log(N)) time, but if you use the remove_task function you mention, and let those zombie tasks accumulate in your queue because you aren't extracting them off the min, then you will see asymptotic slowdown which you wouldn't otherwise see. Of course, you couldn't do this with Queue.PriorityQueue in the first place, so no comparison can be made here.)
The version in the Queue module is implemented using the heapq module, so they have equal efficiency for the underlying heap operations.
That said, the Queue version is slower because it adds locks, encapsulation, and a nice object oriented API.
The priority queue suggestions shown in the heapq docs are meant to show how to add additional capabilities to a priority queue (such as sort stability and the ability to change the priority of a previously enqueued task). If you don't need those capabilities, then the basic heappush and heappop functions will give you the fastest performance.
I was reading about priority queue in python and came across two ways to use them:
-
Heapq module
-
Priority Queue class
In the priority queue class we have 'put' method just like Java's 'offer'. And python has 'get' vs Java's 'poll' method. But i couldn't find any 'peek' method like the one in Java's priority queue. Is it really not implemented. If yes then why?