The expression (sum(row) for row in M) creates what's called a generator. This generator will evaluate the expression (sum(row)) once for each row in M. However, the generator doesn't do anything yet, we've just set it up.

The statement next(G) actually runs the generator on M. So, if you run next(G) once, you'll get the sum of the first row. If you run it again, you'll get the sum of the second row, and so on.

>>> M = [[1,2,3],
...      [4,5,6],
...      [7,8,9]]
>>> 
>>> G = (sum(row) for row in M) # create a generator of row sums
>>> next(G) # Run the iteration protocol
6
>>> next(G)
15
>>> next(G)
24

See also:

• Documentation on generators
• Documentation on yield expressions (with some info about generators)

The special methods __iter__ and __next__ are part of the iterator protocol to create iterator types. For this purpose, you have to differentiate between two separate things: Iterables and iterators.

Iterables are things that can be iterated, usually, these are some kind of container elements that contain items. Common examples are lists, tuples, or dictionaries.

In order to iterate an iterable, you use an iterator. An iterator is the object that helps you iterate through the container. For example, when iterating a list, the iterator essentially keeps track of which index you are currently at.

To get an iterator, the __iter__ method is called on the iterable. This is like a factory method that returns a new iterator for this specific iterable. A type having a __iter__ method defined, turns it into an iterable.

The iterator generally needs a single method, __next__, which returns the next item for the iteration. In addition, to make the protocol easier to use, every iterator should also be an iterable, returning itself in the __iter__ method.

As a quick example, this would be a possible iterator implementation for a list:

class ListIterator:
    def __init__ (self, lst):
        self.lst = lst
        self.idx = 0

    def __iter__ (self):
        return self

    def __next__ (self):
        try:
            item = self.lst[self.idx]
        except IndexError:
            raise StopIteration()
        self.idx += 1
        return item

The list implementation could then simply return ListIterator(self) from the __iter__ method. Of course, the actual implementation for lists is done in C, so this looks a bit different. But the idea is the same.

Iterators are used invisibly in various places in Python. For example a for loop:

for item in lst:
    print(item)

This is kind of the same to the following:

lst_iterator = iter(lst) # this just calls `lst.__iter__()`
while True:
    try:
        item = next(lst_iterator) # lst_iterator.__next__()
    except StopIteration:
        break
    else:
        print(item)

So the for loop requests an iterator from the iterable object, and then calls __next__ on that iterable until it hits the StopIteration exception. That this happens under the surface is also the reason why you would want iterators to implement the __iter__ as well: Otherwise you could never loop over an iterator.

As for generators, what people usually refer to is actually a generator function, i.e. some function definition that has yield statements. Once you call that generator function, you get back a generator. A generator is esentially just an iterator, albeit a fancy one (since it does more than move through a container). As an iterator, it has a __next__ method to “generate” the next element, and a __iter__ method to return itself.

An example generator function would be the following:

def exampleGenerator():
    yield 1
    print('After 1')
    yield 2
    print('After 2')

The function body containing a yield statement turns this into a generator function. That means that when you call exampleGenerator() you get back a generator object. Generator objects implement the iterator protocol, so we can call __next__ on it (or use the the next() function as above):

>>> x = exampleGenerator()
>>> next(x)
1
>>> next(x)
After 1
2
>>> next(x)
After 2
Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    next(x)
StopIteration

Note that the first next() call did not print anything yet. This is the special thing about generators: They are lazy and only evaluate as much as necessary to get the next item from the iterable. Only with the second next() call, we get the first printed line from the function body. And we need another next() call to exhaust the iterable (since there’s not another value yielded).

But apart from that laziness, generators just act like iterables. You even get a StopIteration exception at the end, which allows generators (and generator functions) to be used as for loop sources and wherever “normal” iterables can be used.

The big benefit of generators and their laziness is the ability to generate stuff on demand. A nice analogy for this is endless scrolling on websites: You can scroll down item after after (calling next() on the generator), and every once in a while, the website will have to query a backend to retrieve more items for you to scroll through. Ideally, this happens without you noticing. And that’s exactly what a generator does. It even allows for things like this:

def counter():
    x = 0
    while True:
        x += 1
        yield x

Non-lazy, this would be impossible to compute since this is an infinite loop. But lazily, as a generator, it’s possible to consume this iterative one item after an item. I originally wanted to spare you from implementing this generator as a fully custom iterator type, but in this case, this actually isn’t too difficult, so here it goes:

class CounterGenerator:
    def __init__ (self):
        self.x = 0

    def __iter__ (self):
        return self

    def __next__ (self):
        self.x += 1
        return self.x

What you're talking about is not really up to the next function, but rather up to how each iterator implements its next method. And, while many iterators do "advance" when you call next, they don't have to.

The prototypical iterator is a virtual "pointer" to a position in some iterable, and their next method returns the value the iterator is currently pointing to, and advances it to point to the next value in the iterable.

For example, consider the iterator returned by calling iter on a list. The first call to next(i) returns element 0, then the next call returns element 1, and so on. You could simulate its behavior like this:

class ListIterator(object):
    def __init__(self, lst):
        self.lst = lst
        self.idx = 0
    def next(self):
        try:
            value = self.lst[self.idx]
        except IndexError:
            raise StopIteration
        self.idx += 1
        return value

And obviously that's intended to be the usual case, hence the name next.

But you can easily create an iterator that doesn't work this way—and not just as some pathological type to prove a point, but actual useful iterators.

For example, look at itertools.repeat. The iterator returned by itertools.repeat(10) just gives you 10 each time you call next on it. You could think of it as an iterator over a list of infinite length whose elements are all 10, but that's obviously not how it really works. It actually works like this:

class RepeatIterator(object):
    def __init__(self, value):
        self.value = value
    def next(self):
        return self.value

Other iterators work by computing values on the fly, pulling buffers off a socket, etc. You can always find a way to think of any iterator as a pointer to a position within some kind of virtual sequence, just like that virtual infinite list of 10s, but it's often a stretch, and in those cases it's more helpful to just think about what's really happening—calling the next method.

By the way, this is a lot easier to talk about in Python 3, where you have a next function that calls a __next__ method, rather than a next method. And of course Python 3's way follows the same pattern as ever other special method—iter(i) calls i.__iter__(), getattr(obj, attr) calls obj.__getattr__(attr), and next(i) calls i.__next__().