In this article we’ll learn about generators in Python and how and why to use them.
The three things that you need to understand to master generators are-
- Generator functions itself.
- Generator expressions.
- Python yield statement
Python generator
Generator is a function which returns a generator iterator, where generator iterator is an object that can be iterated.
A generator function in Python looks like a normal function except that it contains yield statement rather than return (it may contain return too but yield statement is must to be qualified as a generator function).
Using yield statement generator generates a value that can be used in a for-loop or that can be retrieved one at a time with the next() function.
Why generator
A normal function acts as a “one call return all” routine which may not be memory efficient when working with large data streams or large files. Consider the following simple example where you have to get all the odd numbers between given range.
def get_odds(start, end):
result = []
for i in range (start, end +1):
if i % 2 != 0:
result.append(i)
return result
As you see here results are appended in the list which is returned at the end of the function execution. When the function is called, it does the execution and returns the whole result.
What if we can lazily return the results on demand? That will definitely make your program more efficient as you won’t be creating a list with all the results at one go. That’s what a Python generator does, when a result is ready it ‘yields’ rather than returning so generator function still maintains its state and next call yields the next number in the series rather than starting from the beginning as is the case with a normal function.
First let’s see a very simple example of Python generator function where numbers 1-3 are generated and returned using yield statements.
def my_generator(): n = 1 yield n n += 1 yield n n += 1 yield n gen = my_generator() print(next(gen)) print(next(gen)) print(next(gen))
Output
1 2 3
As you can see with generator state of local variable is not destroyed as the function call ends, by using next() method you can get the next number.
Now if we write the function to return odd numbers as a generator then it can be written as following.
def get_odds(start, end):
result = []
for i in range (start, end +1):
if i % 2 != 0:
yield(i)
for num in get_odds(1, 100):
print(num)
Here you can see that the generator function is used with a for loop directly, this is possible because generator returns an iterator.
Python generator returns an iterator
As we know generator is a function which returns a generator iterator and that generator iterator is an object. With that knowledge you should keep in mind the following points.
- Since it is an object so you can assign that object reference to a variable.
- Since it is an iterator so you can use next() function or __next__ method to iterate through all the items.
- For loop iterates over an iterable so generator function can be used directly with for loop.
Here is an example of generator function producing odd numbers where returned iterator’s reference is assigned and used.
def get_odds(start, end):
result = []
for i in range (start, end +1):
if i % 2 != 0:
yield(i)
# assigning iterator object reference
gen = get_odds(1, 100)
for num in gen:
print(num)
Here is another example where generator function represents an infinite stream (while True is used) to produce infinite odd numbers. In the example for loop provides the range of displayed odd numbers and next() function gets the next item from the iterator.
def get_odds():
n = 1
while True:
yield n
n += 2
# assigning iterator object reference
gen = get_odds()
for i in range(1, 51):
print(next(gen))
Creating generators using Generator expression
Just like list comprehension is used to quickly create a list using very less code, generator expression creates a generator quickly with very less code.
Generator expressions are syntactically almost identical to list comprehensions, where these two differ is that generator expressions are enclosed in parentheses rather than brackets.
Syntax of Generator expression
(expression for item in iterable) (expression for item in iterable if condition)
Generator expression creation example
Here is an example where both list and a generator are created using list comprehension and a generator expression respectively.
# list comprehension even_list = [num*2 for num in range(1, 6)] print(even_list) # generator expression even_gen = (num*2 for num in range(1, 6)) print(even_gen)
Output
[2, 4, 6, 8, 10] <generator object <genexpr> at 0x0000025BB3E49B88>
As you can see for list the whole list is created with its elements where as generator expression provides a generator object which can give values one at a time using next method.
# generator expression
even_gen = (num*2 for num in range(1, 6))
for i in range(1, 6):
print(next(even_gen))
Output
[2, 4, 6, 8, 10] 2 4 6 8 10
Python yield statement
It’s a yield statement that transforms a normal function into a generator function. In a normal function when the execution finishes control is transferred completely from the function i.e. state is not maintained.
In case of generators state is maintained, whenever generator function encounters a "yield" statement, the generator yields its control, returning a new value from the iterator. When next value is required generator function starts from the previous stored state and returns next value. Theoretically generator can generate infinite values, yielding one value at a time.
send() method in Python generator
Using send() method of the generator you can resume the generator and also send a value back to generator function that is used by the yield statement to return a value. So send() method does three things-
- Resume the generator function
- Send a value to the generator.
- Return the value from the yield statement.
def counter(max):
i = 0
while i < max:
val = (yield i)
# If value is sent, change counter
if val is not None:
i = val
else:
i += 1
gen = counter(10)
print(next(gen))
print(next(gen))
# send new value
print(gen.send(6))
print(next(gen))
Output
0 1 6 7
That's all for this topic Python Generator, Generator Expression, Yield Statement. If you have any doubt or any suggestions to make please drop a comment. Thanks!
>>>Return to Python Tutorial Page
Related Topics
You may also like-