Bite-sized Twisted: Deferreds

In this post, we'll examine the Deferred.

The check's in the mail

A Deferred is kind of like an IOU. It's a promise. It's a forecast of things to come. A forerunner of sorts. A title bearer or trumpeteer.

Enough analogies: it's a class you can instantiate. Like this:

>>> from twisted.internet.defer import Deferred
>>> iou = Deferred()

If you are given a Deferred, use addCallback to register a function to handle the value when it arrives.

>>> def spend(value):
...   print value + ' spent!'
... 
>>> iou.addCallback(spend)
<Deferred at 0xb764bcccL>

If you are the one giving out the Deferred and are responsible for fulfilling the promise, use callback to send the value to the registered callback functions:

>>> iou.callback('$20.00')
$20.00 spent!

Callbacks can be chained

Multiple callbacks can be added to a Deferred. Each callback is not called with the original value. Each callback is called with the return value of the prior callback. For example:

from twisted.internet.defer import Deferred

def mouth(what):
    print 'I am the mouth, I got ' + what
    return 'chewed up ' + what

def stomach(what):
    print 'I am the stomach, I got ' + what
    
d = Deferred()
d.addCallback(mouth)
d.addCallback(stomach)
d.callback('jelly beans')

deferred1.py

I am the mouth, I got jelly beans
I am the stomach, I got chewed up jelly beans

python deferred1.py

Extra arguments

If you want to pass additional arguments to a callback, add them in addCallback:

from twisted.internet.defer import Deferred

def callback(value, arg, keyword_arg='foo'):
    print 'value: %r' % value
    print 'arg: %r' % arg
    print 'keyword_arg: %r' % keyword_arg

d = Deferred()
d.addCallback(callback, 'apple', keyword_arg='banana')
d.callback('gorilla')

deferred2.py

value: 'gorilla'
arg: 'apple'
keyword_arg: 'banana'

python deferred2.py

Read the source

A Deferred works approximately like this PoorMansDeferred:

class PoorMansDeferred:

    def __init__(self):
        self.callbacks = []

    def addCallback(self, function):
        self.callbacks.append(function)

    def callback(self, value):
        for callback in self.callbacks:
            value = callback(value)


def government(value):
    portion = value / 2
    print 'Government got %s' % portion
    return value - portion

def vinny(value):
    portion = value / 2
    print 'Vinny got %s' % portion
    return value - portion

def me(value):
    print 'I got %s' % value

d = PoorMansDeferred()
d.addCallback(government)
d.addCallback(vinny)
d.addCallback(me)
d.callback(30)

not_a_great_deferred.py

Government got 15
Vinny got 7
I got 8

python not_a_great_deferred.py

Vinny should study up on integer division. You should probably study up on self defense.

The real Deferred implementation is not that long and worth reading. It includes error handling, chaining, extra arguments, canceling, etc... Read the source here.

There's no reactor

Please notice a few things about the above examples:

1. The reactor was not involved
2. There was no reactor.run() call
3. reactor wasn't used
4. A Deferred does not need the reactor

In short, a Deferred does not depend on the reactor. It is made more useful by the reactor but it works independently from it.

Many of Twisted's APIs either return or work with Deferreds. It is time well-spent learning how they work.

Defer explosions!

Though a Deferred does not require the reactor, it can be used with it. Let's add an ignite method to our Bomb that returns a Deferred:

from twisted.internet import reactor
from twisted.internet.defer import Deferred

class Bomb:
    
    def __init__(self, fuse, size):
        self.fuse = fuse
        self.size = size

    def ignite(self):
        d = Deferred()
        reactor.callLater(self.fuse, d.callback, self)
        return d

boom/game.py

Then we can modify the Protocol we wrote a few posts ago to use the Deferred code:

from twisted.internet.protocol import Protocol
from twisted.internet import reactor, stdio

from boom.game import Bomb

def explode(bomb):
    print 'BOOM! (magnitude %s)' % bomb.size

class BombProtocol(Protocol):

    def dataReceived(self, data):
        fuse = int(data)
        bomb = Bomb(fuse, fuse)
        bomb.ignite().addCallback(explode)

stdio.StandardIO(BombProtocol())
reactor.run()

async_input3.py

10
3
5
4
2
5
1
BOOM! (magnitude 3)
BOOM! (magnitude 1)
BOOM! (magnitude 2)
BOOM! (magnitude 4)
BOOM! (magnitude 5)
BOOM! (magnitude 5)
BOOM! (magnitude 10)

python async_input3.py

Errors next time

Error handling deserves a post of its own. In fact, that's probably what the next post will be about. Until then, chew on this:

from twisted.internet.defer import Deferred

def toInfinity(value):
    return value / 0

def errorHandler(error):
    print 'There was an error: %s' % error.value

d = Deferred()
d.addCallback(toInfinity)
d.addErrback(errorHandler)
d.callback(293)

print "The error didn't kill the program"

errback1.py

There was an error: integer division or modulo by zero
The error didn't kill the program

python errback1.py

Also consider reading Twisted's official, very thorough guide.