Tornado add_callback considered harmful

July 16, 2018

I ran into an interesting Tornado event loop gotcha last week, so I thought I’d blog the issue. At work I’m building a suite of Tornado Python based microservices to implement an FX options booking stack. The message transports in play for comms with other systems are IBM MQ series and AMPS for pubsub messaging. Naturally we use pymqi, which is a thin Python layer on top of MQ series traditional blocking
C API. Blocking APIs don’t play well with the Tornado event loop. In classic single threaded async style Tornado callbacks should complete quickly and not hog the thread, so that new incoming socket events can be seviced. So the MQ PUTs and GETs were in another thread, which used ioloop.add_callback( ) to
send work back to the main thread. For high volume queues, that means a lot of add_callback( ) invocations. And that’s the gotcha: Tornado’s main event loop dispatches pending timeouts and callbacks before looking for new socket events and dispatching to their handlers. If you have a lot of expensive pending callbacks then your code may be slow to fall through to servicing new socket events. And if those socket events are HTTP GET /health hits from your Marathon platform, then
Marathon will conclude your process is dead and bounce it. Exactly what you don’t want when there’s high traffic! In this case the fix was to make the MQ thread more selective about which incoming messages got pitched over to the other thread with add_callback( ).

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s