Generator Expressions and Iterators in Python

Posted by Mark on November 23, 2022 at 06:33 | Last modified: June 2, 2022 14:08

After doing some further research, I have learned that my previous explanation of initializing multiple variables as empty lists was not very Pythonic. I want to correct some of that today.

The following is a generator expression:

        ( [] for i in range(3) )

The expression will generate three empty lists. Parentheses are needed around generator expressions like brackets are needed around list comprehensions:

A generator is a function that returns an object also known as an iterator.

An iterator is an object that contains a countable number of values. An iterator may be iterated over one value at a time until all values have been exhausted. Trying to iterate further will then raise a StopIteration exception (i.e. error or traceback).

A generator expression is like a list comprehension except the latter produces the entire list while the former produces one item at a time. A generator expression has “lazy execution” (producing items only when asked for), which makes it more memory efficient than a list comprehension.

Although the end result is to initialize three variables as empty lists, Pythonically speaking this is doing something else:

        list_a, list_b, list_c = ( [] for i in range(3) )

The right side is an object containing three items: [], [], and []. The left side unpacks that object into three respective variables. I went into detail about unpacking with regard to the zip() method in these two posts.

Last time, I may have described things as if something magical was going on. This is not magic. The parentheses on the right represent a generator object and the commas on the left are to unpack it.

With regard to my previous discussion about making sure to use commas rather than equals signs, the latter…

        a = b = c = (1, 2, 3)

…is basically doing:

        c = (1, 2, 3)
        b = c
        a = b

b and c both get assigned (1, 2, 3).

With regard to the generator expression from above, this…

        a = b = c = ( [] for i in range(3) )

…is basically doing:

        c = ( [] for i in range(3) )
        b = c
        a = b

Because c cannot be assigned three different lists from the generator, Python assigns the generator to the variable a (and b):

        <generator object <genexpr> at 0x0000019F45352200>

Lists, tuples, dictionaries, and strings are all iterable objects that have an iter() method used to return values. For example:

A for loop creates an iterator object, executes the next() method for each loop, and automatically stops at the end leaving us worry-free with regard to any potential [StopIteration] error: