How can I make a custom data structure as fast as dict?
You can't (at least, not in CPython). CPython's dict is written in C, as are other dict variants in the standard library such as collections.defaultdict. If you want to write custom data structures approaching the speed of a dict, go write it in C, Rust, or run your Python script using PyPy. Anything subclassing a dict becomes an object implemented in Python rather than C, so will not run nearly as fast as a dict.
How can I make my code better?
Your code at the moment is extremely confusing. Here are some of the reasons why:
- You don't have any docstrings. It's really hard, as an outside reader, to understand what's going on in your code if you don't say what a specific function is for, and what its expected behaviour is. Type hints are also very helpful in this regard.
- You don't have any inline comments. Similar to point (1). There's all sorts of things going on in your code inside functions that are frankly a little head-scratching. There's probably a really good reason for a lot of them -- so if there's something that isn't self-explanatory happening inside a function, tell us why it's happening!
- It's really unclear why so many of your methods are private. What's the specific reason that the
_key method is a private method? What would be so terrible about exposing that detail to other users? You have a lot of private methods, and the general expectation in Python is that a method should be public unless there's a good reason why.
- Your attributes are very confusingly named. Your UDict class has one instance attribute
self.Keys, one instance attribute self.KEYS, one instance method self.keys(), and one instance method self._key(). This would be bad even if they recorded similar kinds of information, but they don't! One is a list, one is a dict, one is a method that returns a KeysView, and one is a method that transforms an "unprocessed" key into a "processed" one. How can anybody be expected to remember the difference between all of these?! There's really nothing wrong with being a bit more verbose and giving your attributes and methods more helpful, descriptive names. Your second attempt in your question improves on this a little, but not much. Why are your classes called IString and IDict? How can anybody be expected to know what they do from their names?
- The motivation for your class still seems a little unclear. You say it's a case-insensitive dict. Fine. But you also have added methods that will return the "integer index" of a key or value. It's unclear why these are necessary for this use case. Ditto for your
multiget and multipop methods -- what are these for? And it's unclear why you would implement multiget and multipop, but not multisetdefault and multipopitem. What's the rationale here?
In addition to the above critiques, I would have started in a different place when it comes to solving this problem. My general attitude is that subclassing dict directly works great for when you're extending a dict -- just adding new features onto it, for a specialised purpose. But if you start overriding all sorts of dict dunder methods, I would generally look at inheriting from collections.abc.MutableMapping instead. The great thing about inheriting from MutableMapping is that by implementing just a few abstractmethods that MutableMapping has, you get a whole host of normal dict methods for free. In my example code below, I haven't had to override .setdefault(), .pop(), .popitem(), .get(), .keys(), .values(), .items(), .clear(), .update(), __contains__() or .__eq__(). They all come "for free" because I'm inheriting from MutableMapping. It just ends up being a lot cleaner, in my opinion.
I've also implemented a helper-class, to deal with some of the implementation-logic regarding case-insensitivity. There was just too much going on in your class, in my opinion; it made sense to shunt some of that code elsewhere.
I don't think there's a great deal of speed improvement in my refactoring below. I think in a few situations, my refactoring is a little faster than yours, but I haven't rigorously tested performance.
My code comes out as being a fair bit longer than yours, but I think the main reason for that is simply because I have a lot more comments and docstrings than you did...
from __future__ import annotations
from typing import Any, TypeVar, Optional, Union, NoReturn
from collections.abc import Mapping, MutableMapping, Iterator, Sequence
from abc import abstractmethod
from functools import cache
# For mapping keys and values
_K = TypeVar('_K')
_V = TypeVar('_V')
# Any subclass of _DictBoilerplateBase
_T = TypeVar('_T', bound='_DictBoilerplateBase[Any, Any]')
# For .get() defaults
_D = TypeVar('_D')
class _DictBoilerplateBase(MutableMapping[_K, _V]):
"""Base class for the mapping objects _KeyMapperDict and CaseInsensitiveDict.
Inheriting from MutableMapping
means the following methods are auto-implemented,
both for this base class
and for all classes inheriting from it:
- .setdefault()
- .pop()
- .popitem()
- .get()
- .keys()
- .values()
- .items()
- .clear()
- .update()
- .__contains__()
- .__eq__()
The following abstractmethods need to be implemented by all subclasses of this class:
- .__init__()
- .__getitem__()
- .__setitem__()
- .__delitem__()
"""
__slots__ = 'data'
data: dict[_K, _V] # For type-checkers
@abstractmethod
def __init__(
self,
data: Optional[Mapping[_K, _V]] = None,
/,
**kwargs
) -> None:
if data is not None:
self.update(data)
if kwargs:
self.update(kwargs)
@classmethod
def fromkeys(
cls: type[_T],
keys: Sequence[_K],
default: Optional[_V] = None,
/
) -> _T:
"""Standard alternative constructor for python mapping objects"""
return cls({key: default for key in keys})
def __iter__(self, /) -> Iterator[_K]:
return iter(self.data.keys())
def __len__(self, /) -> int:
return len(self.data)
def __repr__(self, /) -> str:
return f'{type(self).__qualname__}({self.data!r})'
def __str__(self, /) -> str:
return f'{type(self).__name__}({self.data})'
def __or__(self: _T, other: Mapping[_K, _V], /) -> _T:
# (Dicts from python 3.9+ support the union operator)
return type(self)({**self, **other})
def __ror__(self: _T, other: Mapping[_K, _V], /) -> _T:
# (Dicts from python 3.9+ support the union operator)
return type(other)({**other, **self})
def __ior__(self: _T, other: Mapping[_K, _V], /) -> _T:
# (Dicts from python 3.9+ support the union operator)
self.update(other)
return self
def copy(self: _T, /) -> _T:
"""Return a shallow copy of the mapping"""
return type(self)(self)
class _KeyMapperDict(_DictBoilerplateBase[_K, _K]):
"""Helper class for the CaseInsensitiveDict class.
Maps lowercase keys to the original case
in which they were entered in the CaseInsensitiveDict.
Also keeps the processed keys in a separate list,
so that the integer index can be retrieved.
This class knows NOTHING about the values
in the CaseInsensitiveDict.
All dunder methods expect to receive preprocessed keys.
"""
__slots__ = 'keys_list'
def __init__(
self,
data: Optional[Mapping[_K, _K]] = None,
/,
**kwargs
) -> None:
# A map of lowercase-to-originalcase-keys
self.data: dict[_K, _K] = {}
# A list of the processed keys, for integer indexing
self.keys_list: list[_K] = []
super().__init__(data, **kwargs)
@staticmethod
@cache
def process_key(key: _K, /) -> _K:
"""Normalise a key to allow for case-insensitivity when dealing with strings.
Keys that are not strings are returned unchanged.
"""
return key.lower() if isinstance(key, str) else key # type: ignore[return-value]
def register_key(
self,
unprocessed_key: _K,
/
) -> Union[tuple[_K, int], tuple[None, None]]:
"""Determine whether a key is in the mapping, return its integer index if so.
Parameters
----------
unprocessed_key, _K:
A key that has not yet been normalised
according to this class's process_key method.
Return
------
If some version of the key is already in the mapping, returns (_K, int),
a tuple consisting of:
- The key in the form it was in
when it was first entered in the CaseInsensitiveDict.
- The integer index of the key in the mapping.
If the key was not already in the mapping,
returns:
- (None, None)
Examples
--------
>>> d = _KeyMapperDict({'spam': 'SPAM', 'eggs': 'eGgS'})
>>> d.register_key('SPAM')
('SPAM', 0)
>>> d.register_key('Eggs')
('eGgS', 1)
>>> d.register_key('bacon')
(None, None)
"""
# This method is essentially a helper method
# for the __setitem__ of the CaseInsensitiveDict.
processed_key = self.process_key(unprocessed_key)
try:
return self[processed_key], self.keys_list.index(processed_key)
except KeyError as err:
self[processed_key] = unprocessed_key
return None, None
def __setitem__(
self,
processed_key: _K,
unprocessed_key: _K,
/
) -> None:
# This method is the mirror image of __delitem__
self.data[processed_key] = unprocessed_key
self.keys_list.append(processed_key)
def get_original_key(self, unprocessed_key: _K, /) -> _K:
"""Return the original key, as it was first inputted into the mapping.
Parameters
----------
unprocessed_key, _K:
A key that has not yet been normalised
according to this class's process_key method.
Return
------
original_key, _K:
The equivalent key to the inputted value,
as it was originally inputted into the mapping.
Raises
------
KeyError if no equivalent
for the inputted key exists in this mapping.
Example
-------
>>> d = _KeyMapperDict({'spam': 'SPAM', 'eggs': 'eGgS'})
>>> d.get_original_key('SPAM')
'SPAM'
>>> d.get_original_key('Eggs')
'eGgS'
>>> d.get_original_key('bacon')
Traceback (most recent call last):
KeyError: 'bacon'
"""
return self[self.process_key(unprocessed_key)]
def __getitem__(self, processed_key: _K, /) -> _K:
return self.data[processed_key]
def remove_key(self, unprocessed_key: _K, /) -> int:
"""Delete the key from the mapping, return the index where the key used to be.
Example
-------
>>> d = _KeyMapperDict({'spam': 'SPAM', 'eggs': 'eGgS', 'bacon': 'bacon'})
>>> d.remove_key('EGGS')
1
>>> d
_KeyMapperDict({'spam': 'SPAM', 'bacon': 'bacon'})
>>> d.remove_key('Spam')
0
>>> d
_KeyMapperDict({'bacon': 'bacon'})
"""
# This method is essentially a helper method
# for the __delitem__ of the CaseInsensitiveDict
processed_key = self.process_key(unprocessed_key)
index = self.keys_list.index(processed_key)
del self[processed_key]
return index
def __delitem__(self, processed_key: _K, /) -> None:
# This method is the mirror image of __setitem__
del self.data[processed_key]
self.keys_list.remove(processed_key)
def index_of_key(self, unprocessed_key: _K, /) -> Optional[int]:
"""Return the "integer index" of a certain key in the keys_list.
Parameters
----------
unprocessed_key, _K:
A key that may or may not be in the keys_list,
and may or may not be of the same case as it was
originally entered into the mapping.
Return
------
index, int or None:
Either the integer index of the key in the keys_list,
or None if the key is not in the keys_list.
Example
-------
>>> d = _KeyMapperDict({'spam': 'SPAM', 'eggs': 'eGgS'})
>>> d.index_of_key('SPAM')
0
>>> d.index_of_key('spam')
0
>>> d.index_of_key('eggs')
1
>>> d.index_of_key('bacon') is None
True
"""
processed_key = self.process_key(unprocessed_key)
try:
return self.keys_list.index(processed_key)
except ValueError:
return None
def original_key_at_index(self, index: int, /) -> Optional[_K]:
"""Return the key at a certain integer index in the keys_list.
Return the key as it was originally entered into the mapping,
rather than the normalised version of the key.
The keys as they were originally entered
into the CaseInsensitiveDict
are stored as this mapping's values.
Parameters
----------
index, int:
An index that may or may not be valid.
Return
------
key, _K or None:
Either the object at that integer index in the keys_list,
or None if the integer index isn't valid.
Example
-------
>>> d = _KeyMapperDict({'spam': 'SPAM', 'eggs': 'eGgS'})
>>> d.original_key_at_index(0)
'SPAM'
>>> d.original_key_at_index(1)
'eGgS'
>>> d.original_key_at_index(5) is None
True
"""
try:
return self.data[self.keys_list[index]]
except IndexError:
return None
class CaseInsensitiveDict(_DictBoilerplateBase[_K, _V]):
"""A case-insensitive dict.
Where `c = CaseInsensitiveDict()`,
`c['Adele']` returns the same value as `c['adele']` or `c['AdElE']`.
The dict also records the integer index of the keys and values.
The dict preserves the original case
of the first time the key was entered into the mapping.
E.g., it will remember that "Adele"
as first entered into the mapping as "Adele",
even if the value associated with "Adele" in the mapping
is updated using the code `c['adele'] = '21'`.
"""
__slots__ = 'keys_map', 'values_list'
def __init__(
self,
data: Optional[Mapping[_K, _V]] = None,
/,
**kwargs
) -> None:
# This is where the actual data is stored
# It maps the UNPROCESSED keys to the values
self.data: dict[_K, _V] = {}
# A map of PROCESSED keys to the ORIGINAL keys
self.keys_map: _KeyMapperDict[_K] = _KeyMapperDict()
# A list of the key values, for integer indexing
self.values_list: list[_V] = []
super().__init__(data, **kwargs)
def __setitem__(self, unprocessed_key: _K, value: _V, /) -> None:
# (original_key, index) will be (_K, int)
# if the key's already in the mapping.
# Else (None, None)
original_key, index = self.keys_map.register_key(unprocessed_key)
# Better to check the index,
# as None can plausibly be used as a dictionary key
if index is None:
self.values_list.append(value)
self.data[unprocessed_key] = value
else:
self.values_list[index] = value
self.data[original_key] = value # type: ignore[index]
def __getitem__(self, unprocessed_key: _K, /) -> _V:
# Catch KeyErrors and reraise them,
# to make for a more logical traceback.
try:
processed_key = self.keys_map.get_original_key(unprocessed_key)
except KeyError as err:
raise KeyError(*err.args) from err
else:
return self.data[processed_key]
def __delitem__(self, unprocessed_key: _K, /) -> None:
# Catch KeyErrors and reraise them,
# to make for a more logical traceback.
try:
original_key = self.keys_map.get_original_key(unprocessed_key)
except KeyError as err:
raise KeyError(*err.args) from err
else:
del self.data[original_key]
# the original_key is not processed -
# the _KeyMapperDict does that for us
self.values_list.pop(self.keys_map.remove_key(original_key))
def index_of_key(self, unprocessed_key: _K, /) -> Optional[int]:
"""Return the integer index of a certain key in the mapping.
This method is effectively delegated
to the instance's `keys_map` attribute,
which is of type `_KeyMapperDict`.
Parameters
----------
unprocessed_key, _K:
A key that may or may not be in the mapping.
Return
------
index, int or None:
Either the integer index of the key in the mapping,
or None if the key is not in the mapping.
Example
-------
>>> d = CaseInsensitiveDict({'a': 'spam', 'b': 'eggs'})
>>> d.index_of_key('A')
0
>>> d.index_of_key('a')
0
>>> d.index_of_key('b')
1
>>> d.index_of_key('c') is None
True
"""
return self.keys_map.index_of_key(unprocessed_key)
def index_of_value(self, *args: Any, **kwargs: Any) -> NoReturn:
"""Raise NotImplementedError.
Example
-------
>>> d = d = CaseInsensitiveDict({'a': 'spam', 'b': 'eggs'})
>>> d.index_of_value('spam')
Traceback (most recent call last):
NotImplementedError: index_of_value \
is deliberately not implemented as a method. \
Multiple values can exist in a dictionary \
that are all the same, therefore it does not make sense \
to request the index of a dictionary value.
"""
raise NotImplementedError(
"index_of_value is deliberately not implemented as a method. "
"Multiple values can exist in a dictionary that are all the same, "
"therefore it does not make sense "
"to request the index of a dictionary value."
)
def index_of_item(self, *args: Any, **kwargs: Any) -> NoReturn:
"""Raise NotImplementedError.
Example
-------
>>> d = d = CaseInsensitiveDict({'a': 'spam', 'b': 'eggs'})
>>> d.index_of_item('a', 'spam')
Traceback (most recent call last):
NotImplementedError: index_of_item is deliberately not implemented. \
Use index_of_key instead.
>>> d.index_of_item(('a', 'spam'))
Traceback (most recent call last):
NotImplementedError: index_of_item is deliberately not implemented. \
Use index_of_key instead.
"""
raise NotImplementedError(
"index_of_item is deliberately not implemented. "
"Use index_of_key instead."
)
def key_at_index(self, index: int, /) -> Optional[_K]:
"""Return the key at a certain integer index in the mapping.
Return the key as it was originally entered into the mapping,
not the normalised version of the key.
This method is effectively delegated
to the instance's `keys_map` attribute,
which is of type `_KeyMapperDict`.
Parameters
----------
index, int:
An index that may or may not be valid.
Return
------
key, _K or None:
Either the key at that integer index in the mapping,
or None if the integer index isn't valid.
Example
-------
>>> d = CaseInsensitiveDict({'a': 'spam', 'b': 'eggs'})
>>> d.key_at_index(0)
'a'
>>> d.key_at_index(1)
'b'
>>> d.key_at_index(2) is None
True
"""
return self.keys_map.original_key_at_index(index)
def value_at_index(self, index: int, /) -> Optional[_V]:
"""Return the value at a certain integer index in the mapping.
Parameters
----------
index, int:
An index that may or may not be valid.
Return
------
value, _V or None:
Either the value at that integer index in the mapping,
or None if the integer index isn't valid.
Example
-------
>>> d = CaseInsensitiveDict({'a': 'spam', 'b': 'eggs'})
>>> d.value_at_index(0)
'spam'
>>> d.value_at_index(1)
'eggs'
>>> d.value_at_index(5) is None
True
"""
try:
return self.values_list[index]
except IndexError:
return None
def item_at_index(self, index: int, /) -> Optional[tuple[_K, _V]]:
"""Return the (key, value) pair at a certain integer index in the mapping.
Parameters
----------
index, int:
An index that may or may not be valid.
Return
------
item, (_K, _V) or None:
Either the (key, value) pair
at that integer index in the mapping,
or None if the integer index isn't valid.
Example
-------
>>> d = CaseInsensitiveDict({'a': 'spam', 'b': 'eggs'})
>>> d.item_at_index(0)
('a', 'spam')
>>> d.item_at_index(1)
('b', 'eggs')
>>> d.item_at_index(2) is None
True
"""
key = self.key_at_index(index)
return None if key is None else (key, self[key])
def multiget(
self,
/,
*key_sequence: _K,
default: Optional[_D] = None
) -> list[Union[_V, _D, None]]:
"""Return a list of values corresponding to an arbitrary sequence of keys.
If any item in key_sequence is not a valid key,
the default value goes in the list.
Example
-------
>>> d = CaseInsensitiveDict({'a': 1, 'c': 2})
>>> d.multiget('a', 'b', 'c', default='spam')
[1, 'spam', 2]
>>> d
CaseInsensitiveDict({'a': 1, 'c': 2})
"""
# The type-hint for the default is _D rather than _V,
# as dict.get() does not update the mapping at all.
# Therefore the default does not need to be the same type
# as the mapping's values.
return [self.get(k, default) for k in key_sequence]
def multipop(self, /, *key_sequence: _K) -> list[_V]:
"""Return a list of values corresponding to an arbitrary sequence of keys.
Remove said keys from the mapping.
Example
-------
>>> d = CaseInsensitiveDict({'a': 1, 'c': 2})
>>> d.multipop('a', 'c')
[1, 2]
>>> d
CaseInsensitiveDict({})
"""
return [self.pop(k) for k in key_sequence]
def multipopitem(self, n: int, /) -> list[tuple[_K, _V]]:
"""Return a list, of length `n`, of (key, value) pairs, popped from the beginning of the mapping.
NOTE: classes inheriting from collections.abc.MutableMapping
pop mapping items in "first-in-first-out" (FIFO) order.
This is the opposite order to python's builtin `dict`,
`collections.OrderedDict` and `collections.Counter`,
all of which use "last-in-first-out" (LIFO).
Example
-------
>>> d = CaseInsensitiveDict({'a': 1, 'b': 2, 'c': 3, 'd': 4})
>>> d.multipopitem(3)
[('a', 1), ('b', 2), ('c', 3)]
>>> d
CaseInsensitiveDict({'d': 4})
"""
return [self.popitem() for _ in range(n)]
def multisetdefault(
self,
/,
*key_sequence: _K,
default: Optional[_V] = None
) -> list[Optional[_V]]:
"""Return a list of values corresponding to an arbitrary sequence of keys.
If any item in key_sequence is not a valid key,
the default value goes in the list.
Additionally, the mapping will be updated
such that the key is added to the mapping,
with this function's `default` parameter
as the associated value.
Example
-------
>>> d = CaseInsensitiveDict({'a': 1, 'c': 2})
>>> d.multisetdefault('a', 'b', 'c', default='spam')
[1, 'spam', 2]
>>> d
CaseInsensitiveDict({'a': 1, 'c': 2, 'b': 'spam'})
"""
return [self.setdefault(k, default) for k in key_sequence] # type: ignore[arg-type]