NumPy arrays are stored as contiguous blocks of memory. They usually have a single datatype (e.g. integers, floats or fixed-length strings) and then the bits in memory are interpreted as values with that datatype.
Creating an array with dtype=object is different. The memory taken by the array now is filled with pointers to Python objects which are being stored elsewhere in memory (much like a Python list is really just a list of pointers to objects, not the objects themselves).
Arithmetic operators such as * don't work with arrays such as ar1 which have a string_ datatype (there are special functions instead - see below). NumPy is just treating the bits in memory as characters and the * operator doesn't make sense here. However, the line
np.array(['avinash','jay'], dtype=object) * 2
works because now the array is an array of (pointers to) Python strings. The * operator is well defined for these Python string objects. New Python strings are created in memory and a new object array with references to the new strings is returned.
If you have an array with string_ or unicode_ dtype and want to repeat each string, you can use np.char.multiply:
In [52]: np.char.multiply(ar1, 2)
Out[52]: array(['avinashavinash', 'jayjay'],
dtype='<U14')
NumPy has many other vectorised string methods too.
Answer from Alex Riley on Stack OverflowVideos
NumPy arrays are stored as contiguous blocks of memory. They usually have a single datatype (e.g. integers, floats or fixed-length strings) and then the bits in memory are interpreted as values with that datatype.
Creating an array with dtype=object is different. The memory taken by the array now is filled with pointers to Python objects which are being stored elsewhere in memory (much like a Python list is really just a list of pointers to objects, not the objects themselves).
Arithmetic operators such as * don't work with arrays such as ar1 which have a string_ datatype (there are special functions instead - see below). NumPy is just treating the bits in memory as characters and the * operator doesn't make sense here. However, the line
np.array(['avinash','jay'], dtype=object) * 2
works because now the array is an array of (pointers to) Python strings. The * operator is well defined for these Python string objects. New Python strings are created in memory and a new object array with references to the new strings is returned.
If you have an array with string_ or unicode_ dtype and want to repeat each string, you can use np.char.multiply:
In [52]: np.char.multiply(ar1, 2)
Out[52]: array(['avinashavinash', 'jayjay'],
dtype='<U14')
NumPy has many other vectorised string methods too.
There are 3 main dtypes to store strings in numpy:
object: Stores pointers to Python objectsstr: Stores fixed-width stringsnumpy.types.StringDType(): New in numpy 2.0 and stores variable-width strings
str consumes more memory than object; StringDType is better
Depending on the length of the fixed-length string and the size of the array, the ratio differs but as long as the longest string in the array is longer than 2 characters, str consumes more memory (they are equal when the longest string in the array is 2 characters long). For example, in the following example, str consumes almost 8 times more memory.
On the other hand, the new (in numpy>=2.0) numpy.dtypes.StringDType stores variable width strings, so consumes much less memory.
from pympler.asizeof import asizeof
ar1 = np.array(['this is a string', 'string']*1000, dtype=object)
ar2 = np.array(['this is a string', 'string']*1000, dtype=str)
ar3 = np.array(['this is a string', 'string']*1000, dtype=np.dtypes.StringDType())
asizeof(ar2) / asizeof(ar1) # 7.944444444444445
asizeof(ar3) / asizeof(ar1) # 1.992063492063492
For numpy 1.x, str is slower than object
For numpy>=2.0.0, str is faster than object
Numpy 2.0 has introduced a new numpy.strings API that has much more performant ufuncs for string operations. A simple test (on numpy 2.2.0) below shows that vectorized string operations on an array of str or StringDType dtype is much faster than the same operations on an object dtype array.
import timeit
t1 = min(timeit.repeat(lambda: ar1*2, number=1000))
t2a = min(timeit.repeat(lambda: np.strings.multiply(ar2, 2), number=1000))
t2b = min(timeit.repeat(lambda: np.strings.multiply(ar3, 2), number=1000))
print(t2a / t1) # 0.8786601958427778
print(t2b / t1) # 0.7311586933668037
t3 = min(timeit.repeat(lambda: np.array([s.count('i') for s in ar1]), number=1000))
t4a = min(timeit.repeat(lambda: np.strings.count(ar2, 'i'), number=1000))
t4b = min(timeit.repeat(lambda: np.strings.count(ar3, 'i'), number=1000))
print(t4a / t3) # 0.13328748153237377
print(t4b / t3) # 0.3365874412749679
For numpy<2.0.0 (tested on numpy 1.26.0)
Numpy 1.x's vectorized string methods are not optimized, so operating on the object array is often faster. For example, in the example in the OP where each character is repeated, a simple * (aka multiply()) is not only more concise but also over 10 times faster than char.multiply().
import timeit
setup = "import numpy as np; from __main__ import ar1, ar2"
t1 = min(timeit.repeat("ar1*2", setup, number=1000))
t2 = min(timeit.repeat("np.char.multiply(ar2, 2)", setup, number=1000))
t2 / t1 # 10.650433758517027
Even for functions that cannot be readily be applied on the array, instead of the vectorized char method of str arrays, it is faster to loop over the object array and work on the Python strings.
For example, iterating over the object array and calling str.count() on each Python string is over 3 times faster than the vectorized char.count() on the str array.
f1 = lambda: np.array([s.count('i') for s in ar1])
f2 = lambda: np.char.count(ar2, 'i')
setup = "import numpy as np; from __main__ import ar1, ar2, f1, f2, f3"
t3 = min(timeit.repeat("f1()", setup, number=1000))
t4 = min(timeit.repeat("f2()", setup, number=1000))
t4 / t3 # 3.251369161574832
On a side note, if it comes to explicit loop, iterating over a list is faster than iterating over a numpy array. So in the previous example, a further performance gain can be made by iterating over the list
f3 = lambda: np.array([s.count('i') for s in ar1.tolist()])
# ^^^^^^^^^ <--- convert to list here
t5 = min(timeit.repeat("f3()", setup, number=1000))
t3 / t5 # 1.2623498005294627
If your Kernel class has a predictable amount of member data, then you could define a dtype for it instead of a class. e.g. if it's parameterized by 9 floats and an int, you could do
kerneldt = np.dtype([('myintname', np.int32), ('myfloats', np.float64, 9)])
arr = np.empty(dims, dtype=kerneldt)
You'll have to do some coercion to turn them into objects of class Kernel every time you want to manipulate methods of a single kernel but that's one way to store the actual data in a NumPy array. If you want to only store a reference, then the object dtype is the best you can do without subclassing ndarray.
It has to be a Numpy scalar type:
http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html#arrays-scalars-built-in
or a subclass of ndarray:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html#numpy.ndarray
This is a bug in the lib.
dtype objects can be constructed dynamically. And NumPy does so all the time. There's no guarantee anywhere that they're interned, so constructing a dtype that already exists will give you the same one.
On top of that, np.float64 isn't actually a dtype; it's a… I don't know what these types are called, but the types used to construct scalar objects out of array bytes, which are usually found in the type attribute of a dtype, so I'm going to call it a dtype.type. (Note that np.float64 subclasses both NumPy's numeric tower types and Python's numeric tower ABCs, while np.dtype of course doesn't.)
Normally, you can use these interchangeably; when you use a dtype.type—or, for that matter, a native Python numeric type—where a dtype was expected, a dtype is constructed on the fly (which, again, is not guaranteed to be interned), but of course that doesn't mean they're identical:
>>> np.float64 == np.dtype(np.float64) == np.dtype('float64')
True
>>> np.float64 == np.dtype(np.float64).type
True
The dtype.type usually will be identical if you're using builtin types:
>>> np.float64 is np.dtype(np.float64).type
True
But two dtypes are often not:
>>> np.dtype(np.float64) is np.dtype('float64')
False
But again, none of that is guaranteed. (Also, note that np.float64 and float use the exact same storage, but are separate types. And of course you can also make a dtype('f8'), which is guaranteed to work the same as dtype(np.float64), but that doesn't mean 'f8' is, or even ==, np.float64.)
So, it's possible that constructing an array by explicitly passing np.float64 as its dtype argument will mean you get back the same instance when you check the dtype.type attribute, but that isn't guaranteed. And if you pass np.dtype('float64'), or you ask NumPy to infer it from the data, or you pass a dtype string for it to parse like 'f8', etc., it's even less likely to match. More importantly, you definitely not get np.float64 back as the dtype itself.
So, how should it be fixed?
Well, the docs define what it means for two dtypes to be equal, and that's a useful thing, and I think it's probably the useful thing you're looking for here. So, just replace the is with ==:
if isinstance(xx_, numpy.ndarray) and xx_.dtype == numpy.float64 and xx_.flags.contiguous:
However, to some extent I'm only guessing that's what you're looking for. (The fact that it's checking the contiguous flag implies that it's probably going to go right into the internal storage… but then why isn't it checking C vs. Fortran order, or byte order, or anything else?)
I'm not sure when this API was introduced, but at least as of 2022 it looks like you can use numpy.issubdtype for the type checking part and therefore write:
if isinstance(arr, numpy.ndarray) and numpy.issubdtype(arr.dtype, numpy.floating):
...