As others have stated, sys.getsizeof only returns the size of the object structure that represents your data. So if, for instance, you have a dynamic array that you keep adding elements to, sys.getsizeof(my_array) will only ever show the size of the base DynamicArray object, not the growing size of memory that its elements take up.
pympler.asizeof.asizeof() gives an approximate complete size of objects and may be more accurate for you.
from pympler import asizeof
asizeof.asizeof(my_object) # should give you the full object size
As others have stated, sys.getsizeof only returns the size of the object structure that represents your data. So if, for instance, you have a dynamic array that you keep adding elements to, sys.getsizeof(my_array) will only ever show the size of the base DynamicArray object, not the growing size of memory that its elements take up.
pympler.asizeof.asizeof() gives an approximate complete size of objects and may be more accurate for you.
from pympler import asizeof
asizeof.asizeof(my_object) # should give you the full object size
sys.getsizeof returns a number which is more specialized and less useful than people think. In fact, if you increase the number of attributes to six, your test3_obj remains at 32, but test4_obj jumps to 48 bytes. This is because getsizeof is returning the size of the PyObject structure implementing the type, which for test3_obj doesn't include the dict holding the attributes, but for test4_obj, the attributes aren't stored in a dict, they are stored in slots, so they are accounted for in the size.
But a class defined with __slots__ takes less memory than a class without, precisely because there is no dict to hold the attributes.
Why override __sizeof__? What are you really trying to accomplish?
BUG: Incorrect results from `sys.getsizeof()` for multi-dimensional arrays
Strange behavior of sys.getsizeof
[Python 3.5] What does sys.getsizeof(object) show me exactly when I use this instruction?
What is the difference between len() and sys.getsizeof() methods in python? - Stack Overflow
Videos
I was watching a numpy video on YouTube and the presenter made a point about numpy arrays versus python lists, and he did so in an odd manner. He was pointing out that one of the advantages of numpy array are how they take up less space, but when I tried to remake his demonstration in IDLE I didn’t get the same results.
import numpy as np
import sys
a = range(1000)
print('Information pertaining to a')
print('Get size', sys.getsizeof(a))
print('Type', type(a))
print('Print of actual a', a, '\n')
b = []
for i in range (1000):
b.append(i)
print('Information pertaining to b')
print('Get size', sys.getsizeof(b))
print('Type', type(b))
print('Print of actual b', b, '\n')
c = np.arange(1000)
print('Information pertaining to c')
print('Get size', sys.getsizeof(c))
print('Type', type(c))
print('Print of actual c', c, '\n')
d = 5
print('Information pertaining to d')
print('Get size', sys.getsizeof(d))
print('Type', type(d))
print('Print of actual d', d, '\n')
e = 'e'
print('Information pertaining to e')
print('Get size', sys.getsizeof(e))
print('Type', type(e))
print('Print of actual e', e, '\n')When I run this code, it shows me that the np array is indeed a bit lighter than the py list (9000 bytes vs 8000 bytes), but for some reason it doesn’t show me the full size of the range. It looks like he’s only showing me the space taken up by the letter 'a' (only 48 bytes). So I’m wondering, what exactly is getsizeof() meant to do? And why does it treat different kinds of list-like objects differently?
They are not the same thing at all.
len() queries for the number of items contained in a container. For a string that's the number of characters:
Return the length (the number of items) of an object. The argument may be a sequence (string, tuple or list) or a mapping (dictionary).
sys.getsizeof() on the other hand returns the memory size of the object:
Return the size of an object in bytes. The object can be any type of object. All built-in objects will return correct results, but this does not have to hold true for third-party extensions as it is implementation specific.
Python string objects are not simple sequences of characters, 1 byte per character.
Specifically, the sys.getsizeof() function includes the garbage collector overhead if any:
getsizeof()calls the object’s__sizeof__method and adds an additional garbage collector overhead if the object is managed by the garbage collector.
String objects do not need to be tracked (they cannot create circular references), but string objects do need more memory than just the bytes per character. In Python 2, __sizeof__ method returns (in C code):
Py_ssize_t res;
res = PyStringObject_SIZE + PyString_GET_SIZE(v) * Py_TYPE(v)->tp_itemsize;
return PyInt_FromSsize_t(res);
where PyStringObject_SIZE is the C struct header size for the type, PyString_GET_SIZE basically is the same as len() and Py_TYPE(v)->tp_itemsize is the per-character size. In Python 2.7, for byte strings, the size per character is 1, but it's PyStringObject_SIZE that is confusing you; on my Mac that size is 37 bytes:
>>> sys.getsizeof('')
37
For unicode strings the per-character size goes up to 2 or 4 (depending on compilation options). On Python 3.3 and newer, Unicode strings take up between 1 and 4 bytes per character, depending on the contents of the string.
For containers such as dictionaries or lists that reference other objects, the memory size given covers only the memory used by the container and the pointer values used to reference those other objects. There is no straightforward method of including the memory size of the ‘contained’ objects because those same objects could have many more references elsewhere and are not necessarily owned by a single container.
The documentation states it like this:
Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to.
If you need to calculate the memory footprint of a container and anything referenced by that container you’ll have to use some method of traversing to those contained objects and get their size; the documentation points to a recursive recipe.
key difference is that len() will give actual length of elements in container , Whereas sys.getsizeof() will give it's memory size which it occupy
for more information read docs of python which is available at https://docs.python.org/3/library/sys.html#module-sys
From the documentation (my bold) (a):
Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to.
So the size of v does not include the sizes of the elements it refers to.
If you change kite into kites, you'll also see that its size increases but not the size of v (I've replaced your big number with 100...00 in the output to ease formatting):
1 size is: 12
2 size is: 12
kite size is: 25
100...00 size is: 102
Total size is: 48
1 size is: 12
2 size is: 12
kites size is: 26
100...00 size is: 102
Total size is: 48
Think of it like this:
/ +-----+
| v | ref | -> 1
Size | | ref | -> 2
of v | | ref | -> 'kite'
| | ref | -> 100**100
\ +-----+
\___________________________/
Size of things referred
to by v
(a) That page also has a link to a recipe for doing recursive size calculations if you need that information. The link is duplicated here for citation, and the code is duplicated below to make this answer more self-contained.
Plugging your structure into that code gives:
48 <type 'list'> [1, 2, 'kites', 100...00L]
12 <type 'int'> 1
12 <type 'int'> 2
26 <type 'str'> 'kites'
102 <type 'long'> 100...00L
200
The code, with your structure, is shown below.
from __future__ import print_function
from sys import getsizeof, stderr
from itertools import chain
from collections import deque
try:
from reprlib import repr
except ImportError:
pass
def total_size(o, handlers={}, verbose=False):
""" Returns the approximate memory footprint an object and all of its contents.
Automatically finds the contents of the following builtin containers and
their subclasses: tuple, list, deque, dict, set and frozenset.
To search other containers, add handlers to iterate over their contents:
handlers = {SomeContainerClass: iter,
OtherContainerClass: OtherContainerClass.get_elements}
"""
dict_handler = lambda d: chain.from_iterable(d.items())
all_handlers = {tuple: iter,
list: iter,
deque: iter,
dict: dict_handler,
set: iter,
frozenset: iter,
}
all_handlers.update(handlers) # user handlers take precedence
seen = set() # track which object id's have already been seen
default_size = getsizeof(0) # estimate sizeof object without __sizeof__
def sizeof(o):
if id(o) in seen: # do not double count the same object
return 0
seen.add(id(o))
s = getsizeof(o, default_size)
if verbose:
print(s, type(o), repr(o), file=stderr)
for typ, handler in all_handlers.items():
if isinstance(o, typ):
s += sum(map(sizeof, handler(o)))
break
return s
return sizeof(o)
##### Example call #####
if __name__ == '__main__':
v = [1,2,'kites',100**100]
print(total_size(v, verbose=True))
This happens because your "Total size" is actually the size of the list structure without the contents. So you can store an object of any size there and it won't change your "Total size." You need a "recursive" getsizeof(), and for that, see here: Python deep getsizeof list with contents? or here: Deep version of sys.getsizeof
Hi Pyople!
Yesterday I learned about sys.getsizeof() function and try some code. More specifically:
lst = [i for i in range(1000000000)] # one mld numbers, creating for about a minute
When I use sys.getsizeof(lst), it returns: 8058558880. Which is correct. But when I look at my system resources in Linux Centos7 IPython (Python 3.4) I see: ipython Memory: 39592564 K Shared Mem: 5176 K - That's freaking 40GB.
I don't understand why, if a object is 8 GB in size, takes 40 KGB system memory. I tried it in list that had around 400 MB and system took 400 * 5 (approx) = 2 GB (approx)
Why is it taking 5-times more memory than it should? Or is the problem only because I tried it in iPython / Konsole? And in program it wouldn't be a problem?
sys.getsizeof gives you the amount of memory allocated to the list itself, but you also have 10...00 int objects that the list only contains a pointer to.
The size of an object does not include the size of all the objects that that object refers to. For example:
>>> import sys
>>> foo = ['a' * 1000000]
>>> sys.getsizeof(foo)
40
>>> sys.getsizeof(foo[0])
1000025
foo is a list object that contains one item. Its size is 40 bytes, because that's how much memory it takes to store a list big enough to hold a reference to one object. That object happens to be about a megabyte in size, but it's a completely separate object from the list object and doesn't count towards the size of the list object.
You misunderstand what sys.getsizeof() does. It returns the amount of memory Python uses for a string object, not length of the line.
Python string objects track reference counts, the object type and other metadata together with the actual characters, so 2978 bytes is not the same thing as the string length.
See the stringobject.h definition of the type:
typedef struct {
PyObject_VAR_HEAD
long ob_shash;
int ob_sstate;
char ob_sval[1];
/* Invariants:
* ob_sval contains space for 'ob_size+1' elements.
* ob_sval[ob_size] == 0.
* ob_shash is the hash of the string or -1 if not computed yet.
* ob_sstate != 0 iff the string object is in stringobject.c's
* 'interned' dictionary; in this case the two references
* from 'interned' to this object are *not counted* in ob_refcnt.
*/
} PyStringObject;
where PyObject_VAR_HEAD is defined in object.h, where the standard ob_refcnt, ob_type and ob_size fields are all defined.
So a string of length 2957 takes 2958 bytes (string length + null) and the remaining 20 bytes you see are to hold the reference count, the type pointer, the object 'size' (string length here), the cached string hash and the interned state flag.
Other object types will have different memory footprints, and the exact sizes of the C types used differ from platform to platform as well.
A string object representing 2957 bytes of data takes more than 2957 bytes of memory to represent, due to overhead such as the type pointer and the reference count. sys.getsizeof includes this additional overhead.