Based on this StackOverflow answer:

NumPy does not support jagged arrays natively. gives an array that may or may not behave as you expect.

A workaround using masked arrays can be as follows:

import numpy as np
import numpy.ma as ma

a = np.array([0, 1])
b = np.array([2, 3, 4, 5])
c = np.array([6, 7, 8, 9, 10, 11])

jagged_array = ma.vstack(
    [
        ma.array(np.resize(a, c.shape[0]), mask=[False, False, True, True, True, True]),
        ma.array(
            np.resize(b, c.shape[0]), mask=[False, False, False, False, True, True]
        ),
        c,
    ]
)
print(jagged_array)
print(jagged_array.ndim)
print(jagged_array.shape)

Your output would look like:

❯ python3 sample.py
[[0 1 -- -- -- --]
 [2 3 4 5 -- --]
 [6 7 8 9 10 11]]
2
(3, 6)

Arrays of arrays are generally inefficient because the sub array are object causing Numpy to fallback on a slow path interacting with the interpreter (due to reference counting, checks, indirections, etc.) or to implicitly convert them into a big array internally (which is generally not possible with jagged arrays). AFAIK, your Boost code should also interact with the interpreter internally in this case. In fact An array of arrays is generally slower than a list of array because Numpy arrays are not built-ins type so CPython needs to calls Numpy functions doing many checks over and over. Additionally, Numpy does not support jagged array natively (see this related post).

Question 1: Is this the proper way of handling jagged arrays?

This is the usual way but clearly not a fast way. An efficient way to encore jagged array is to concatenate them in a big 1D array and use an additional array to store the start/stop indices (or offset/size informations). That way enable you to still use some basic Numpy vectorized methods on all arrays of sub-slices. Numba can be used to speed up the iteration over sub-arrays. The same thing applies for C++ with Boost.

Question 2: I create an object np::ndarray row to index into the second dimensions. This seems a potential performance bottleneck. Can that somehow be avoided? Can I use somehow the length of the 2nd arrays and index directly into the data buffer? I am not sure if there is padding or if there is a contiguous block of memory behind a jagged array? I assume it is not.

AFAIK, the performance bottleneck is due to the interaction with the CPython interpreter (via the CPython API) so to deal with CPython object. The above solution solve this problem since you only need to read an integer from the slicing array. The jagged array can be seen as a custom type or as a simple tuple of two arrays (possibly three if you want to split the start/stop or offsets/size). This representation is a bit similar to sparse matrices.

Note that arrays of arrays objects are indeed not stored contiguously in memory (each array object is stored are a different location in memory independently of other arrays that is dependant of the underlying CPython allocator). Arrays of arrays objects are typically stored as a pointer to an object structure containing a pointer to a memory buffer that contains pointers to objects containing each a pointer to other memory buffers. This cause a lot of pointer indirections and thus bad performance.

You might think, your input is a list of tuples. However, it is a list of integers and tuples. (880) will be interpreted as an integer, but not as a tuple. So you have to deal with both datatypes.

First of all I suggest converting your input data to a list of lists. Each of the lists contained in that list should have the same length, because an array supports constant dimensions only. Therefore, I would convert the elements into a list and fill missing values with zeros (to make all elements equal in length).

If we do this for all of the elements given in the input list, we create a new list containing lists of equal length which can be converted into an array.

A very basic (and error-prone) approach would look like this:

import numpy as np


original_list = [
    (880),
    (880, 1080),
    (880, 1080, 1080),
    (470, 470, 470, 1250),
]


def get_len(item):
    try:
        return len(item)
    except TypeError:
        # `(880)` will be interpreted as an int instead of a tuple
        # so we need to handle tuples and integers
        # as integers do not support len(), a TypeError will be raised
        return 1


def to_list(item):
    try:
        return list(item)
    except TypeError:
        # `(880)` will be interpreted as an int instead of a tuple
        # so we need to handle tuples and integers
        # as integers do not support __iter__(), a TypeError will be raised
        return [item]


def fill_zeros(item, max_len):
    item_len = get_len(item)
    to_fill = [0] * (max_len - item_len)
    as_list = to_list(item) + to_fill
    return as_list


max_len = max([get_len(item) for item in original_list])
filled = [fill_zeros(item, max_len) for item in original_list]

arr = np.array(filled)
print(arr)

Printing:

[[ 880    0    0    0]
[ 880 1080    0    0]
[ 880 1080 1080    0]
[ 470  470  470 1250]]

Short answer: you can't. NumPy does not support jagged arrays natively.

Long answer:

>>> a = ones((3,))
>>> b = ones((2,))
>>> c = array([a, b])
>>> c
array([[ 1.  1.  1.], [ 1.  1.]], dtype=object)

gives an array that may or may not behave as you expect. E.g. it doesn't support basic methods like sum or reshape, and you should treat this much as you'd treat the ordinary Python list [a, b] (iterate over it to perform operations instead of using vectorized idioms).

Several possible workarounds exist; the easiest is to coerce a and b to a common length, perhaps using masked arrays or NaN to signal that some indices are invalid in some rows. E.g. here's b as a masked array:

>>> ma.array(np.resize(b, a.shape[0]), mask=[False, False, True])
masked_array(data = [1.0 1.0 --],
             mask = [False False  True],
       fill_value = 1e+20)

This can be stacked with a as follows:

>>> ma.vstack([a, ma.array(np.resize(b, a.shape[0]), mask=[False, False, True])])
masked_array(data =
 [[1.0 1.0 1.0]
 [1.0 1.0 --]],
             mask =
 [[False False False]
 [False False  True]],
       fill_value = 1e+20)

(For some purposes, scipy.sparse may also be interesting.)

Your array is 2x2:

In [298]: A
Out[298]: 
array([[array([1, 2, 3]), array([4, 5])],
       [array([6, 7, 8, 9]), array([10])]], dtype=object)

While A+A works, boolean tests have not been implemented for this kind of array:

In [299]: A>4
...
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I'm going to flatten A because it makes it easier to compare with list operations:

In [301]: A1=A.flatten()

In [303]: A1+A1
Out[303]: 
array([array([2, 4, 6]), array([ 8, 10]), array([12, 14, 16, 18]),
       array([20])], dtype=object)

In [304]: [a+a for a in A1]
Out[304]: [array([2, 4, 6]), array([ 8, 10]), array([12, 14, 16, 18]), array([20])]

In [305]: timeit A1+A1
100000 loops, best of 3: 6.85 µs per loop

In [306]: timeit [a+a for a in A1]
100000 loops, best of 3: 9.09 µs per loop

The array operation is a bit faster than a list comprehension. But if I first turn the array into a list:

In [307]: A1l=A1.tolist()

In [308]: A1l
Out[308]: [array([1, 2, 3]), array([4, 5]), array([6, 7, 8, 9]), array([10])]

In [309]: timeit [a+a for a in A1l]
100000 loops, best of 3: 5.2 µs per loop

times improve. This is a good indication that the A1+A1 (or even A+A) is using a similar sort of iteration.

So the straight forward way of performing your A,B calculation is

In [310]: A2=[a[a>4] for a in A1]
In [311]: B=[a+a for a in A2]
In [312]: B
Out[312]: [array([], dtype=int32), array([10]), array([12, 14, 16, 18]), array([20])]

(we can convert to/from arrays and lists as needed).

A numpy array stores its data a flat databuffer, and uses the shape and strides attributes to quickly calculate the location of any element, regardless of the dimensions. The fast array operations use compiled code that rapidly steps though the databuffers of arguments, performing the operations element by element (or some other combination).

A dtype object array also has the flat databuffer, but the elements are pointers to lists or arrays elsewhere. So while it can index individual elements quickly, it still has to perform a Python call(s) to access the arrays. So especially when the array is 1d, it is virtually the same as a flat list with the same pointers.

Multidimensional object arrays are nicer than nested lists. You can reshape them, access elements (A[1,3] v Al[1][3]), transpose them, etc. But when it comes to iterating through all the subarrays they don't offer much of a benefit.

Looking again at your 2d array:

In [315]: timeit A+A
100000 loops, best of 3: 6.93 µs per loop  # 6.85 for A1+A1 (above)

In [316]: timeit [[j+j for j in i] for i in A]
100000 loops, best of 3: 17.1 µs per loop

In [317]: Al = A.tolist()

In [318]: timeit [[j+j for j in i] for i in Al]
100000 loops, best of 3: 7.01 µs per loop    # 5.2 for A1l flat list

Basically the same time for summing the array and iterating through the equivalent nested list.

You can transform your initial array of iterable-objects to ndarray by padding them with zeros in a vectorized manner:

import numpy as np

a = np.array([[1, 2, 3, 4], 
              [2, 3, 4], 
              [4, 5, 6]])
max_len = len(max(a, key = lambda x: len(x))) # max length of iterable-objects contained in array
cust_func = np.vectorize(pyfunc=lambda x: np.pad(array=x, 
                                                 pad_width=(0,max_len), 
                                                 mode='constant', 
                                                 constant_values=(0,0))[:max_len], otypes=[list])
a_pad = np.stack(cust_func(a))

output:

array([[1, 2, 3, 4],
       [2, 3, 4, 0],
       [4, 5, 6, 0]])