Note: this post assumes Python 3.x syntax.^†

A generator is simply a function which returns an object on which you can call next, such that for every call it returns some value, until it raises a StopIteration exception, signaling that all values have been generated. Such an object is called an iterator.

Normal functions return a single value using return, just like in Java. In Python, however, there is an alternative, called yield. Using yield anywhere in a function makes it a generator. Observe this code:

>>> def myGen(n):
...     yield n
...     yield n + 1
... 
>>> g = myGen(6)
>>> next(g)
6
>>> next(g)
7
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

As you can see, myGen(n) is a function which yields n and n + 1. Every call to next yields a single value, until all values have been yielded. for loops call next in the background, thus:

>>> for n in myGen(6):
...     print(n)
... 
6
7

Likewise there are generator expressions, which provide a means to succinctly describe certain common types of generators:

>>> g = (n for n in range(3, 5))
>>> next(g)
3
>>> next(g)
4
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Note that generator expressions are much like list comprehensions:

>>> lc = [n for n in range(3, 5)]
>>> lc
[3, 4]

Observe that a generator object is generated once, but its code is not run all at once. Only calls to next actually execute (part of) the code. Execution of the code in a generator stops once a yield statement has been reached, upon which it returns a value. The next call to next then causes execution to continue in the state in which the generator was left after the last yield. This is a fundamental difference with regular functions: those always start execution at the "top" and discard their state upon returning a value.

There are more things to be said about this subject. It is e.g. possible to send data back into a generator (reference). But that is something I suggest you do not look into until you understand the basic concept of a generator.

Now you may ask: why use generators? There are a couple of good reasons:

• Certain concepts can be described much more succinctly using generators.
• Instead of creating a function which returns a list of values, one can write a generator which generates the values on the fly. This means that no list needs to be constructed, meaning that the resulting code is more memory efficient. In this way one can even describe data streams which would simply be too large to fit in memory.

• Generators allow for a natural way to describe infinite streams. Consider for example the Fibonacci numbers:

  >>> def fib():
  ...     a, b = 0, 1
  ...     while True:
  ...         yield a
  ...         a, b = b, a + b
  ... 
  >>> import itertools
  >>> list(itertools.islice(fib(), 10))
  [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
  

  This code uses itertools.islice to take a finite number of elements from an infinite stream. You are advised to have a good look at the functions in the itertools module, as they are essential tools for writing advanced generators with great ease.

  ^† About Python <=2.6: in the above examples next is a function which calls the method __next__ on the given object. In Python <=2.6 one uses a slightly different technique, namely o.next() instead of next(o). Python 2.7 has next() call .next so you need not use the following in 2.7:

>>> g = (n for n in range(3, 5))
>>> g.next()
3

Quick answer:

Doing list() around a generator expression is (almost) exactly equivalent to having [] brackets around it. So yeah, you can do

>>> list((x for x in string.letters if x in (y for y in "BigMan on campus")))

But you can just as well do

>>> [x for x in string.letters if x in (y for y in "BigMan on campus")]

Yes, that will turn the generator expression into a list comprehension. It's the same thing and calling list() on it. So the way to make a generator expression into a list is to put brackets around it.

In Python 3, you could unpack the generator expression into a print statement:

>>> print(*(x for x in string.ascii_letters if x in (y for y in "BigMan on campus")))
a c g i m n o p s u B M

Detailed explanation:

A generator expression is a "naked" for expression. Like so:

x*x for x in range(10)

Now, you can't stick that on a line by itself, you'll get a syntax error. But you can put parenthesis around it.

>>> (x*x for x in range(10))
<generator object <genexpr> at 0xb7485464>

This is sometimes called a generator comprehension, although I think the official name still is generator expression, there isn't really any difference, the parenthesis are only there to make the syntax valid. You do not need them if you are passing it in as the only parameter to a function for example:

>>> sorted(x*x for x in range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Basically all the other comprehensions available in Python 3 and Python 2.7 is just syntactic sugar around a generator expression. Set comprehensions:

>>> {x*x for x in range(10)}
{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}

>>> set(x*x for x in range(10))
{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}

Dict comprehensions:

>>> dict((x, x*x) for x in range(10))
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

>>> {x: x*x for x in range(10)}
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

And list comprehensions under Python 3:

>>> list(x*x for x in range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

>>> [x*x for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Under Python 2, list comprehensions is not just syntactic sugar. But the only difference is that x will under Python 2 leak into the namespace.

>>> x
9

While under Python 3 you'll get

>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

This means that the best way to get a nice printout of the content of your generator expression in Python is to make a list comprehension out of it! However, this will obviously not work if you already have a generator object. Doing that will just make a list of one generator:

>>> foo = (x*x for x in range(10))
>>> [foo]
[<generator object <genexpr> at 0xb7559504>]

In that case you will need to call list():

>>> list(foo)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Although this works, but is kinda stupid:

>>> [x for x in foo]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

You want you use yield from:

import itertools

sequence = ['a', 'b', 'c', 'd']

def next_player(seq):
    yield from itertools.cycle(seq)

g = next_player(sequence)
for _ in range(6):
    print(next(g))

Output:

a
b
c
d
a
b

To elaborate: yield from can chain interables, in this case it's the same as writing:

def next_player(seq):
    for x in itertools.cycle(seq):
        yield x

but since itertools.cycle already returns a generator-like object, you could just write

def next_player(seq):
    return itertools.cycle(seq)