Lazy Looping Helpers

Iterators are Everywhere

Many of Python built-in functions return iterators instead of lists.

The enumerate and reversed functions return iterators. Files in Python are also iterators.

>>> letters = ['a', 'b']
>>> next(enumerate(letters))
(0, 'a')
>>> next(reversed(letters))
'b'
>>> next(open('widgets.txt'))
'WIDGET READING 1\n'

All of these iterators act like lazy iterables. They don’t do any work, until you start looping over them.

In Python 2, many functions like range returned lists; if you wanted an iterator, you had to use a special version of the function. In Python 3, the special versions like xrange are removed and wherever it makes sense, iterators are returned instead.

Many functions built-in to Python also accept generic iterables, including lazy iterators.

For example enumerate and zip both accept any kind of iterables, including iterators:

>>> numbers = [1, 2, 3]
>>> squares = (n**2 for n in numbers)
>>> squares
<generator object <genexpr> at 0x100583b10>
>>> zip(numbers, squares)
<zip object at 0x1005ce2c8>
>>> list(zip(numbers, squares))
[(1, 1), (2, 4), (3, 9)]

Iterators in the Standard Library

The standard library includes a number of functions which are meant to work with lazy iterables as well as functions that return lazy iterables, usually in the form of iterators.

The itertools library includes many such functions:

The itertools.count function produces continuous incremental numbers forever.

If you want to print out every number for the rest of time, do this:

>>> from itertools import count
>>> for x in count():
>>>     print(x)
...

We could get an infinite list of square numbers like this:

>>> squares = (n**2 for n in count())

You have to be careful with count, or it will go on forever. It is important to make sure that there is some way of stopping the loop.

If we wanted just the first 10 squares we could try to slice this generator:

>>> squares[:10]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'generator' object is not subscriptable

But we’ll get an error because generators are not indexable and therefore not sliceable.

We could use itertools.islice to slice any iterable instead:

>>> from itertools import islice
>>> squares = (n**2 for n in count())
>>> list(islice(squares, 10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

We could also use itertools.takewhile to get all the squares which are less than 1000:

>>> from itertools import takewhile
>>> squares = (n**2 for n in count())
>>> def less_than_1000(n): return n < 1000
...
>>> list(takewhile(less_than_1000, squares))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961]

Embracing Laziness

Anytime you come across something that creates an entire list, only to loop through the list and throw it away later, consider lazy iterables. Either with generator expressions or generator functions, you can potentially save space and improve the efficiency of a program by making the looping lazy.

itertools Exercises

Except as noted, these exercises are in the iteration.py file in the exercises directory. Edit the file to add the functions or fix the error(s) in the existing function(s).

To run the test: from the exercises folder, type python test.py <function_name>, like this:

$ python test.py get_primes_over

All Together

Edit the all_together function we saw before in generators.py so that it takes any number of iterables and strings them together. Use something from itertools to solve this exercise.

Example:

>>> from generators import all_together
>>> list(all_together([1, 2], (3, 4), "hello"))
[1, 2, 3, 4, 'h', 'e', 'l', 'l', 'o']
>>> nums = all_together([1, 2], (3, 4))
>>> list(all_together(nums, nums))
[1, 2, 3, 4]

Total Length

Edit the total_length function so that it calculates the total length of all given iterables.

Example:

>>> from iteration import lotal_length
>>> total_length([1, 2, 3])
3
>>> total_length()
0
>>> total_length([1, 2, 3], [4, 5], iter([6, 7]))
7

lstrip

Edit the lstrip function to accept an iterable and an item to strip from the beginning of the iterable. The lstrip function should return an iterator which, when looped over, will provide each of the items in the given iterable after all of the strip values have been removed from the beginnign.

Example:

>>> list(lstrip([0, 0, 1, 0, 2, 3, 0], 0))
[1, 0, 2, 3, 0]
>>> ''.join(lstrip('hhello there' 'h'))
'ello there'

Stop On

Edit the stop_on function we saw before in functions.py so that it accepts an iterable and a value and yields from the given iterable repeatedly until the given value is reached. Use something from itertools to solve this exercise.

Example:

>>> from functions import stop_on
>>> list(stop_on([1, 2, 3], 3))
[1, 2]
>>> next(stop_on([1, 2, 3], 1), 0)
0

Interleave

Edit the interleave function we saw before in the generators.py file so that it works with iterables of different length. Any short iterables should be skipped over once exhausted.

For example:

>>> list(interleave([1, 2, 3], [6, 7, 8, 9]))
[1, 6, 2, 7, 3, 8, 9]
>>> list(interleave([1, 2, 3], [4, 5, 6, 7, 8]))
[1, 4, 2, 5, 3, 6, 7, 8]
>>> list(interleave([1, 2, 3, 4], [5, 6]))
[1, 5, 2, 6, 3, 4]

Note: to test this exercise you’ll need to comment out the appropriate @unittest.skip line in generators_test.py.

Big Primes

Edit the get_primes_over function we saw before in the functions.py file so that it yields a specified number of prime numbers greater than 1,000,000.

Try doing this without using while loops or for loops, using itertools.

You can use this function to determine whether a number is prime:

def is_prime(candidate):
    """Return True if candidate is a prime number"""
    for n in range(2, candidate):
        if candidate % n == 0:
            return False
    return True
✕
↑
Write more Pythonic code

I send out 1 Python exercise every week through a Python skill-building service called Python Morsels.

If you'd like to improve your Python skills every week, sign up!

You can find the Privacy Policy here.
reCAPTCHA protected (Google Privacy Policy & TOS)