Thursday, July 5, 2012

Bite-sized Twisted:Telnet

Finally, in this post, we'll monitor keypresses instead of lines of text. This means that you won't have to press enter after each command.

Strictly speaking, the changes we'll make to the code are for Telnet, and not really Twisted-specific -- though Twisted's support for Telnet makes it easier.

Telnet's modes

On my computer (running Ubuntu), running telnet will default to line by line mode. I can see this by starting the obstinate server:

from twisted.internet.protocol import Protocol, Factory
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor

class ObstinateProtocol(Protocol):

def dataReceived(self, data):
self.transport.write('no\r\n')

factory = Factory()
factory.protocol = ObstinateProtocol
endpoint = TCP4ServerEndpoint(reactor, 8900)
endpoint.listen(factory)
reactor.run()
obstinate1.py

Then if I start telnet and send some stuff, I see that it reacts to the lines I send:

$ telnet 127.0.0.1 8900
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Wanna play?
no
Are you obstinate?
no
Are you a duck?
no

We can switch the telnet client to character mode by typing the escape character Control ], then typing mode char. From then on, the server will receive each byte as we type it:

^]mode char

telnet> mode char
no
no
no
no
no
no

Each of those no lines was printed after pressing single keys. Great! We don't have to press enter anymore to play boom!

You can force the client to character mode

by telling the Telnet client that you will echo, will supress go-ahead and won't use linemode. We'll use Twisted's Telnet protocol from the conch package (changed lines are highlighted):

from twisted.internet.protocol import Protocol, Factory
from twisted.conch.telnet import Telnet, SGA, LINEMODE, ECHO
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor

class ObstinateProtocol(Telnet):

def connectionMade(self):
self.will(ECHO)
self.will(SGA)
self.wont(LINEMODE)

def enableLocal(self, option):
return option in (ECHO, SGA)

def applicationDataReceived(self, data):
self.transport.write('no\r\n')


factory = Factory()
factory.protocol = ObstinateProtocol
endpoint = TCP4ServerEndpoint(reactor, 8900)
endpoint.listen(factory)
reactor.run()
lessobstinate.py

Telnet into this and you should see the same behavior as when we set the client to character mode manually.

Finally, a playable game

I've pushed changes to the tx-telnet-2 branch on GitHub. You can see everything that's changed since the last post here.

To play the game, get the tx-telnet-2 branch and run run.py as before. Then telnet to port 8900 multiple times from multiple computers. If you die, you'll have to telnet in again :)

Thursday, June 21, 2012

Bite-sized Twisted:Networking Triforce

This post begins a series of posts about networking -- an area where Twisted shines.

Network Input/Output == Bytes

In the second Bite-sized Twisted post we accepted input (bytes) from stdin asynchronously. In this post, we'll accept input (bytes) over the network. I'm intentionally emphasizing bytes (bytes). It's all just bytes.

It might be worth reviewing that second post, so the concepts are fresh.

Parts of the Triforce

In Twisted, dealing with networking (moving bytes) usually involves three things:

  1. Protocol - determines fate of bytes
  2. Transport - carries bytes between protocol and remote
  3. Factory - creates protocols

Protocol

A Protocol's job is to do application-specific things with bytes. Code for deciding what bytes should be sent back across the wire is also in the Protocol. When it decides to send bytes across the wire, it will invoke the Transport's write method.

Here's an ObstinateProtocol which sends the bytes 'no\r\n' in response to all data received:

from twisted.internet.protocol import Protocol

class ObstinateProtocol(Protocol):

def dataReceived(self, data):
self.transport.write('no\r\n')
protocol1.py

Transport

As seen above, the Transport provides an interface for interacting with the remote side. It has a write method which sends bytes to the remote side. There is generally one Transport per Protocol. You can access the transport through a Protocol's transport attribute.

Factory

A Factory's job is to make a Protocol for a connection. In general, there is one Factory for many Protocol instances.

Standard Factories determine what Protocol to make based on their protocol attribute. This snippet shows a Factory that will make ObstinateProtocol protocols:

from twisted.internet.protocol import Factory
factory = Factory()
factory.protocol = ObstinateProtocol
factory1.py

Notice that factory.protocol is set to the class ObstinateProtocol and not an instance ObstinateProtocol().

Magic glue

One other component is needed for the Networking Triforce to work: magic glue. Here's a complete example that serves the ObstinateProtocol using TCP on port 8900 (magic glue is highlighted):

from twisted.internet.protocol import Factory, Protocol
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ServerEndpoint

class ObstinateProtocol(Protocol):

def dataReceived(self, data):
self.transport.write('no\n')

factory = Factory()
factory.protocol = ObstinateProtocol
endpoint = TCP4ServerEndpoint(reactor, 8900)
endpoint.listen(factory)
reactor.run()
glue1.py

Start glue1.py in one shell, then open another shell and start telnet with this command:

telnet 127.0.0.1 8900

You've just connected to the obstinate server. Type something then press enter; you should see 'no' written back. For example:

$ telnet 127.0.0.1 8900
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Are you there?
no
Are you lying?
no
Are you sure?
no

If you have another computer nearby, start glue1.py here, then start telnet on the other computer, changing 127.0.0.1 to this computer's IP (you can even do this while the other telnet session is still going).

A thousand words to explain the magic glue

I don't like magical code. So, here's the pertinent code for hooking up the Triforce (from twisted/internet/tcp.py). First, the factory is asked to build a protocol:

protocol = self.factory.buildProtocol(self._buildAddr(addr))
portion of twisted/internet/tcp.py

Factory.buildProtocol looks like this:

class Factory:

def buildProtocol(self, addr):
p = self.protocol()
p.factory = self
return p
portion of twisted/internet/protocol.py

Then a transport is created:

transport = self.transport(skt, protocol, addr, self, s, self.reactor)
portion of twisted/internet/tcp.py

The transport is connected to the protocol:

protocol.makeConnection(transport)
portion of twisted/internet/tcp.py

which looks like this:

class BaseProtocol:

def makeConnection(self, transport):
self.connected = 1
self.transport = transport
portion of twisted/internet/protocol.py

Here's a thousand words describing it:

Toward a better boom

Here's a Protocol and Factory that will let you telnet in to play.

from twisted.internet.protocol import Protocol, Factory
import string


from boom.game import Pawn, YoureDead, IllegalMove


class SimpleProtocol(Protocol):

num = 0

move_mapping = {
'w': 'u',
'a': 'l',
'd': 'r',
's': 'd',
}

def connectionMade(self):
self.factory.protocols.append(self)
name = string.uppercase[self.num % len(string.uppercase)]
SimpleProtocol.num += 1
self.pawn = Pawn(name)
self.factory.board.insertPawn((0,0), self.pawn)


def connectionLost(self, reason):
self.factory.protocols.remove(self)
self.factory.board.pawns.remove(self.pawn)


def dataReceived(self, data):
for k in data:
if k in self.move_mapping:
try:
self.pawn.move(self.move_mapping[k])
except YoureDead, e:
pass
except IllegalMove, e:
pass
elif k == 'e':
try:
self.pawn.dropBomb()
except YoureDead, e:
pass
except IllegalMove, e:
pass



class SimpleFactory(Factory):
"""
A factory for making L{SimpleProtocol}s

@ivar board: the game board on which I'll be playing
@ivar protocols: A list of L{SimpleProtocol} instances currently
in use.
"""

protocol = SimpleProtocol


def __init__(self, board):
self.board = board
self.protocols = []
boom/protocol.py

The code is available from GitHub in the tx-telnet-1 branch. Do this:

git clone -b tx-telnet-1 git://github.com/iffy/boom boom.git
cd boom.git
PYTHONPATH=. python run.py

Or if you don't have Git:

wget https://github.com/iffy/boom/tarball/tx-telnet-1
tar xf tx-telnet-1
cd iffy-boom-f5c6433/
PYTHONPATH=. python run.py

Then connect with:

telnet 127.0.0.1 8900

Next time, we'll remove the need to press enter after each move. In the meantime, have a look at Twisted's guide to writing servers for more information.

Monday, June 18, 2012

Bite-sized Twisted: Asynchronous Testing and the Clock

In this, the seventh Bite-sized Twisted post, I'll go over a little asynchronous testing and the Clock.

Impatient runners

Let's test that a lit bomb explodes with a message when it should. Or in other words, that a function returns a Deferred which fires after a time. For convenience, I'm going to put the code to be tested in the same file as the test.

We'll need to import some things:

from unittest import TestCase
from twisted.internet import reactor
from twisted.internet.defer import Deferred
test_bomb1.py:1-3

Here's the test:

class BombTest(TestCase):

def test_explodes(self):
def check(result):
self.assertEqual(result, 'exploded')
d = lightBomb(2, 'exploded')
d.addCallback(check)
test_bomb1.py:5-11

And here's the function:

def lightBomb(fuse, message):
defer = Deferred()
reactor.callLater(fuse, defer.callback, message)
return defer
test_bomb1.py:12-16

When we run it, we see that it passes the test:

test_bomb1
BombTest
test_explodes ... [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 0.001s

PASSED (successes=1)
trial test_bomb1.py

Yay!

No, not yay. Change the test to this:

    def test_explodes(self):
def check(result):
print "I was executed with %r" % result
d = lightBomb(2, 'exploded')
d.addCallback(check)
test_bomb2.py:7-11

And see that check is never called:

test_bomb2
BombTest
test_explodes ... [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 0.001s

PASSED (successes=1)
trial test_bomb2.py

What gives?

Wait up!

Does it seem a little suspicious that the bomb is supposed to go off in 2 seconds and the test runs in less than 1 second? The problem is that the test runner is not waiting for the Deferred to be called back. To wait for the Deferred, we'll have to use twisted.trial.unittest.TestCase and return the Deferred from the test method:

from twisted.trial.unittest import TestCase
from twisted.internet import reactor
from twisted.internet.defer import Deferred
test_bomb3.py:1-3
    def test_explodes(self):
def check(result):
self.assertEqual(result, 'exploded')
d = lightBomb(2, 'exploded')
d.addCallback(check)
return d
test_bomb3.py:7-12
test_bomb3
BombTest
test_explodes ... [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 2.012s

PASSED (successes=1)
trial test_bomb3.py

See how the test took more than 2 seconds to run this time?

Timeout

What if lightBomb looked like this instead?

def lightBomb(fuse, message):
defer = Deferred()
return defer
test_hang.py:14-16

Run the test and it will hang. Add a timeout to fail any test that takes too long:

class BombTest(TestCase):

timeout = 3
test_timeout.py:5-7
test_timeout
BombTest
test_explodes ... [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: twisted.internet.defer.TimeoutError: <test_timeout.BombTest testMethod=test_explodes> (test_explodes) still running at 3.0 secs

test_timeout.BombTest.test_explodes
-------------------------------------------------------------------------------
Ran 1 tests in 3.015s

FAILED (errors=1)
trial test_timeout.py

Hello, Clock

If you have a lot of tests dealing with scheduled events and timing, running all of them will take a long time. To avoid that, we can simulate the passage of time with the Clock.

Here's the same test as test_bomb3.py using the Clock (changed lines are highlighted):

from twisted.trial.unittest import TestCase
from twisted.internet.task import Clock
from twisted.internet import reactor
from twisted.internet.defer import Deferred

class BombTest(TestCase):

def test_explodes(self):
def check(result):
self.assertEqual(result, 'exploded')
clock = Clock()
d = lightBomb(2, 'exploded', reactor=clock)
d.addCallback(check)
clock.advance(2)
return d

def lightBomb(fuse, message, reactor=reactor):
defer = Deferred()
reactor.callLater(fuse, defer.callback, message)
return defer
test_clock.py
test_clock
BombTest
test_explodes ... [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 0.005s

PASSED (successes=1)
trial test_clock.py

Yes! Much faster! Here are the Clock docs if you are interested. Note in test_clock.py how we had to make lightBomb accept an optional reactor.

Boom tests

All of the code for the current version of boom was test-driven. Perhaps reading some of the tests will be enlightening -- though I make no claim at being a testing expert. The test code is available on GitHub.

Thursday, June 14, 2012

Bite-sized Twisted: A simple, working program

There's an asynchronous, single-player Bomberman clone at the end of this post.

Quick overview

Very quickly, these are the components of the game:

  1. players (the things running around dropping bombs)
  2. bombs
  3. fire (from the bombs)
  4. the game board
This post will build them in the reverse order: the board, fire, bombs then players. I'm going to skim through the normal Python, only emphasizing the points where the Twisted library does something neat.

Coordinates throughout the program are x,y tuples (e.g. (3,2)).

Big fat disclaimer

I'm not a game programmer. I've previously written simple games like Tetris. I know there are cool optimizations that game makers use (for dealing with lag and minimizing network traffic and such). I'm not going to prematurely optimize. I will accept good patches though :)

Da board

The game board will be divided into layers. At the bottom is the bottom layer... which, for now, consists of identical background tiles. One layer up is the layer of destructable and indestructable tiles. We'll call these the foreground tiles. Here's how we'll implement the foreground tiles and a method to generate a basic board:

EMPTY = 0
HARD = 1
SOFT = 2

class Board:

def __init__(self):
self.fg_tiles = {}

def generate(self, width, height):
for x in xrange(width):
for y in xrange(height):
coord = (x,y)
if (x % 2) and (y % 2):
self.fg_tiles[coord] = HARD
else:
self.fg_tiles[coord] = SOFT
board1.py

EMPTY means there's no foreground tile, SOFT tiles are destructable and HARD tiles are indestructable. A basic 5 x 5 board looks like this:

+-----+
|:::::|
|:#:#:|
|:::::|
|:#:#:|
|:::::|
+-----+

: = SOFT
# = HARD

The keys for fg_tiles are coordinate tuples (e.g. (4,5)). There's nothing asynchronous or Twisted-y about that code. Let's move along.

Start the fire!

Generally, an exploding bomb will make fire appear on the board. Once a bomb makes the fire, however, the bomb has nothing more to do with it. Let's implement the post-bomb portion of the fire.

We'll keep track of fires in a dictionary on Board. The keys into the fires dictionary are x,y coordinate tuples:

    def __init__(self):
self.fg_tiles = {}
self.fires = {}
fire1.py

Fires have an asynchronous component: they ignite, burn for some amount of time, then die. We'll use reactor.callLater to do the "burn for some amount time" bit and use a Deferred to signal to the fire-starter when the fire is extinguished:

from twisted.internet import reactor
from twisted.internet.defer import Deferred

...

def startFire(self, coord, burntime):
defer = Deferred()

# Put the fire out after a bit.
reactor.callLater(burntime, self.stopFire, coord)

# Record that there's a fire on the board.
self.fires[coord] = defer

# Destroy whatever tile is here
self.fg_tiles[coord] = EMPTY
return defer


def stopFire(self, coord):
defer = self.fires[coord]

# Remove the fire from the board
del self.fires[coord]

# notify people who care that the fire is out
defer.callback(None)
fire2.py

The Deferred is created, stored in the fires dictionary, then returned to the caller (they can add callbacks to it if they want). Also, the tile at that coordinate is "destroyed" (made EMPTY).

The callLater calls stopFire, which remove the entry in the fires dictionary and calls back the stored Deferred. That the Deferred is called back with None simply means that any attached callbacks will be passed None as the first argument.

Let's talk about threading. If you've done threaded programming, you may be cringing over the blatant disregard with which global variables are accessed (self.fires and self.fg_tiles) without proper locking mechanisms to guarantee atomicity. This code, though, isn't threaded. startFire will run from beginning to end, sequentially, without any other code executing. Breathe well! This is Twisted!

Restart the fire!

It's possible that an exploding bomb will ignite a tile that is already burning. With fire2.py, that scenario is a problem because the second fire will overwrite the first fire's entry in the fires dict. Then the first fire's stopFire call will delete the entry from fires dict so that when the second fire's stopFire happens, a KeyError is raised. From the players' perspective, the second fire will be prematurely extinguished.

So instead of having two fires burning in the same tile, if a second fire ignites, let's extend the existing fire's time. We can do this with the reset method of the thing returned by callLater (it's called a DelayedCall and you can read the docs here):

    def startFire(self, coord, burntime):
# Is there already a fire on this tile?
if coord in self.fires:
defer, call = self.fires[coord]
# Stoke the fire
call.reset(burntime)
else:
defer = Deferred()
# Put the fire out after a bit.
call = reactor.callLater(burntime, self.stopFire, coord)

# Record that there's a fire on the board.
self.fires[coord] = (defer, call)

# Destroy whatever tile is here
self.fg_tiles[coord] = EMPTY
return defer


def stopFire(self, coord):
d, call = self.fires[coord]

# Remove the fire from the board
del self.fires[coord]

# notify people who care that the fire is out
d.callback(None)
fire3.py

Now a second call to start a fire on a tile will prolong the burning without glitches. Anyone waiting on the Deferred returned by the first call will just wait a little longer.

Bomb, bomb, bomb

Now that we can start fires at will, let's add bombs to the fray. We'll track bombs similarly to fires:

    def __init__(self):
self.fg_tiles = {}
self.bombs = {}
self.fires = {}
bomb1.py

And the code for dropping and detonating bombs is very similar to that for igniting and extinguishing fires (with some extra in detonateBomb to account for the different kinds of tiles and board boundaries, etc...):

    dft_burn = 1


def dropBomb(self, coord, fuse, size):
# Set the bomb up to explode later
defer = Deferred()
call = self._reactor.callLater(fuse, self.detonateBomb, coord)

# Place the bomb on the board
self.bombs[coord] = (defer, call, size)
return defer


def detonateBomb(self, coord):
defer, call, size = self.bombs[coord]

# Remove the bomb from the board
del self.bombs[coord]

# Let people who care know that the bomb has exploded
defer.callback(None)

# Start the fires up, down, left and right
self.startFire(coord, self.dft_burn)
directions = [
(0, -1),
(0, 1),
(-1, 0),
(1, 0),
]
for i in xrange(1, size+1):
for d in list(directions):
target = (coord[0]+(d[0]*i), coord[1]+(d[1]*i))
try:
tile = self.fg_tiles[target]
except KeyError:
continue
if tile == HARD:
directions.remove(d)
continue
if target in self.bombs:
directions.remove(d)
if tile == SOFT:
directions.remove(d)
self.startFire(target, self.dft_burn)
bomb2.py

In Bomberman, exploding bombs cause other bombs to explode. We can incorporate that by changing startFire (changed lines are highlighted):

    def startFire(self, coord, burntime):
# Is there already a fire on this tile?
if coord in self.fires:
defer, call = self.fires[coord]
# Stoke the fire
call.reset(burntime)
else:
defer = Deferred()
# Put the fire out after a bit.
call = reactor.callLater(burntime, self.stopFire, coord)

# Record that there's a fire on the board.
self.fires[coord] = (defer, call)

# Destroy whatever tile is here
self.fg_tiles[coord] = EMPTY

# Is there a bomb to detonate here?
if coord in self.bombs:
self.detonateBomb(coord)
return defer
bomb3.py

But now what happens when the bomb's original fuse expires? It will cause problems trying to re-explode an exploded bomb. So, let's cancel the original fuse using the cancel() method of the DelayedCall:

    def detonateBomb(self, coord):
defer, call, size = self.bombs[coord]

# Remove the bomb from the board
del self.bombs[coord]

# Let people who care know that the bomb has exploded
defer.callback(None)

# Premature explosion?
if call.active():
call.cancel()

# Start the fires up, down, left and right
self.startFire(coord, self.dft_burn)
...
bomb4.py

Now the fuse is canceled on prematurely ignited bombs.

Merely a pawn

I'm using the term pawn to represent the thing that moves around on the board. We'll keep track of Pawns in a set rather than a dictionary because multiple Pawns can be at the same location at once.

class Board:

def __init__(self):
self.fg_tiles = {}
self.bombs = {}
self.fires = {}
self.pawns = set()
pawn1.py

Pawns have various attributes, as shown:

class Pawn:

bombs = 1
flame_size = 1
fuse = 2.0
alive = True

def __init__(self, name=None):
self.name = name
pawn2.py

First things first: Pawns need to know how to die.

    def kill(self):
self.alive = False
pawn3.py

We need to be able to put Pawns on the Board:

class Board:
...
def insertPawn(self, coord, pawn):
# Let the Board and Pawn know about each other
pawn.board = self
pawn.loc = coord
self.pawns.add(pawn)

# Clear a space for the Pawn
self.fg_tiles[pawn.loc] = EMPTY
directions = [
(1,0),
(-1,0),
(0,1),
(0,-1),
]
for d in directions:
target = (pawn.loc[0]+d[0], pawn.loc[1]+d[1])
if target in self.fg_tiles:
self.fg_tiles[target] = EMPTY


class Pawn:
...
board = None
loc = None
pawn4.py

A fire + a Pawn = death for the Pawn:

    def startFire(self, coord, burntime):
# Is there already a fire on this tile?
if coord in self.fires:
defer, call = self.fires[coord]
# Stoke the fire
call.reset(burntime)
else:
defer = Deferred()
# Put the fire out after a bit.
call = reactor.callLater(burntime, self.stopFire, coord)

# Record that there's a fire on the board.
self.fires[coord] = (defer, call)

# Destroy whatever tile is here
self.fg_tiles[coord] = EMPTY

# Is there a bomb to detonate here?
if coord in self.bombs:
self.detonateBomb(coord)

# Any pawns that should diaf?
for pawn in [x for x in self.pawns if x.loc==coord]:
pawn.kill()
return defer
pawn5.py

Pawns can move (very quickly in this version). And when they move, the Board is notified:

class YoureDead(Exception): pass
class IllegalMove(Exception): pass


class Board:
...
def pawnMoved(self, pawn, new_loc):
pawn.loc = new_loc

# Did the Pawn unwittingly move into a fire?
if new_loc in self.fires:
pawn.kill()


class Pawn:
...
def move(self, direction):
if not self.alive:
raise YoureDead("Dead people can't move")

# Which way?
delta = {
'u': (0,-1),
'd': (0,1),
'l': (-1,0),
'r': (1,0),
}[direction]
target = (self.loc[0]+delta[0], self.loc[1]+delta[1])

# Is it okay to move there?
try:
tile = self.board.fg_tiles[target]
except KeyError:
raise IllegalMove("That would be off the board")
if tile != EMPTY:
raise IllegalMove("There's a brick there")
if target in self.board.bombs:
raise IllegalMove("There's a bomb there")

# Go ahead and move
self.board.pawnMoved(self, target)
pawn6.py

Merely an asynchronous pawn

Nothing in the above Pawn snippets makes use of Twisted. But the act of dropping a bomb on the board does:

class Pawn:
...
def dropBomb(self):
if not self.alive:
raise YoureDead("Dead people can't drop bombs")
if self.bombs <= 0:
raise IllegalMove("You're out of bombs")

# Use a bomb from the Pawn's stash
self.bombs -= 1
d = self.board.dropBomb(self.loc, self.fuse, self.flame_size)

# Get the bomb back after it explodes
def bombExploded(result, pawn):
pawn.bombs += 1
d.addCallback(bombExploded, self)
pawn7.py

To keep track of the bombs allotted to the Pawn, we subtract when it's put down, then add when the bomb's Deferred is called back.

Wanna play?

Do this:

git clone -b tx-single-player git://github.com/iffy/boom boom.git
cd boom.git
PYTHONPATH=. python run.py

Or if you don't have Git:

wget https://github.com/iffy/boom/tarball/tx-single-player
tar xf tx-single-player
cd iffy-boom-58c1582/
PYTHONPATH=. python run.py

The controls are W, A, S, D for up, left, down, right respectively. E lays a bomb. AND you have to press enter after each key press. I know, I know! "What a dumb game! Twisted is terrible!" I just haven't covered it yet. We'll get there. Remember, it's bite-sized Twisted.

All the code for this post is in the tx-single-player branch on GitHub. Unlike the snippets shown here, the code there is fully tested and has class and method documentation. Take a look at game.py

Next time

Next time, I'll likely highlight some asynchronous testing tools, then move on to making the game multiplayer and playable (not having to press enter every move).

Tuesday, June 12, 2012

Bite-sized Twisted: Deferred Errors

In this post, I'll talk about the Deferred callback chain, including error handling.

Exceptions

Boring, run-of-the-mill Exception:

def mouth(food):
raise Exception("Don't eat %s" % food)
mouth('cannon balls')
err1.py
Traceback (most recent call last):
File "err1.py", line 3, in <module>
mouth('cannon balls')
File "err1.py", line 2, in mouth
raise Exception("Don't eat %s" % food)
Exception: Don't eat cannon balls
python err1.py

Exception raised inside a callback:

from twisted.internet.defer import Deferred

def mouth(food):
raise Exception("Don't eat %s" % food)

d = Deferred()
d.addCallback(mouth)
d.callback('cannon balls')
err2.py
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "err2.py", line 8, in <module>
d.callback('cannon balls')
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 361, in callback
self._startRunCallbacks(result)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 455, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 542, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "err2.py", line 4, in mouth
raise Exception("Don't eat %s" % food)
exceptions.Exception: Don't eat cannon balls
python err2.py

It complains of an Unhandled error in Deferred. To handle it, an error callback (errback for short) is needed.

from twisted.internet.defer import Deferred

def mouth(food):
raise Exception("Don't eat %s" % food)

def badfood(food):
print "Bad food happened, but I'm okay"

d = Deferred()
d.addCallback(mouth)
d.addErrback(badfood)
d.callback('cannon balls')
err3.py
Bad food happened, but I'm okay
python err3.py

Errbacks are only called to handle errors. So, in this case, the errback is not called since there's no error:

from twisted.internet.defer import Deferred

def mouth(food):
print "I ate %s" % food

def badfood(food):
print "Bad food happened, but I'm okay"

d = Deferred()
d.addCallback(mouth)
d.addErrback(badfood)
d.callback('piano keys')
err4.py
I ate piano keys
python err4.py

Garden of forking paths

Callbacks will always result in either:

  1. a value or
  2. an error
If a value is returned, the next callback in line will get the value. If an error happens, the next errback in line will get the error.

Errbacks are the same -- they will always result in either a value or an error. And the result will be handled either by the next callback or the next errback. This snippet illustrates that:

from twisted.internet.defer import Deferred

def value(result, msg):
print 'value %s' % msg
return 'value'

def error(result, msg):
print 'error %s' % msg
raise Exception(msg)

d = Deferred()
d.addCallback(value, 'callback level 1')

d.addErrback(value, 'errback level 2')

d.addCallback(error, 'callback level 3')
d.addCallback(value, 'callback level 4')
d.addCallback(error, 'callback level 5')
d.addCallback(value, 'callback level 6')

d.addErrback(value, 'errback level 7')
d.addErrback(error, 'errback level 8')
d.addErrback(value, 'errback level 9')
d.addErrback(error, 'errback level 10')

d.addCallback(value, 'callback level 11')

d.callback('begin')
twopaths1.py
value callback level 1
error callback level 3
value errback level 7
value callback level 11
python twopaths1.py

What happened?

  1. The Deferred is called back with 'begin'.
  2. The first callback is called with 'begin' and returns 'value'.
  3. Since there is no error, the errback on level 2 is skipped and the next callback (level 3) is called.
  4. The callback on level 3 ends in error, so all the callbacks are skipped until the next errback (level 7) is found.
  5. The errback on level 7 returns 'value', so all the errbacks are skipped until the next callback (level 11) is found.
Here's a diagram of what happened. Blue squares are callbacks. Red squares are errbacks. Time flows to the right:

1 2 3 4 5 6 7 8 9 10 11

Trapping specific exceptions

Error handlers can be made to only deal with specific exceptions using the trap() method of the passed-in error (the first argument). If the actual error doesn't match the kind being trapped, the next errback in line will be called with the error.

from twisted.internet.defer import Deferred

def add3(result):
return 3 + result

def catchAttributeError(error):
error.trap(AttributeError)
print 'AttributeError happened'
return 'Bad attribute'

def catchTypeError(error):
error.trap(TypeError)
print 'TypeError happened'
return 'Bad type of value'

def printResult(result):
print 'Result: %s' % result

d = Deferred()
d.addCallback(add3)
d.addErrback(catchAttributeError)
d.addErrback(catchTypeError)
d.addCallback(printResult)
d.callback('foo')
trap1.py
TypeError happened
Result: Bad type of value
python trap1.py

Without the .trap() calls in trap1.py, catchAttributeError would have handled the error instead of catchTypeError. Also, notice that catchAttributeError is called first, but since it's only trapping AttributeErrors, the error is passed to the next errback, catchTypeError.

Why can you call .trap() on the thing passed to errbacks? Short answer: it's a Failure object. Long answer: see the "Read more" section below.

Read more

Believe it or not, there's still more you can do with Deferreds. As features are needed for the Bomberman clone, I'll introduce them. In the meantime, read the very thorough, official Deferred Reference if you're interested.

Next time: Bombs away!

Using the things I've posted about so far, I plan for the next Bite-sized Twisted post to include the code for a playable version of boom. Lest you expect more than I'm planning: it will be single player, may or may not display a textual representation of the board, and will require pressing enter after every move. Future posts will improve the game further.