Coding the markets

Early vs late binding

etrading — Sun, 29 Jan 2012 13:56:29 +0000

I agree with pretty much everything Jeff has to say on strongly typed, statically bound languages vs weakly typed late binding systems. But I don’t agree that we must decide in favour of the former. There is no one size fits all language, and I believe the right approach is to use a mix of early and late binding. Early binding when we want efficiency, compactness and speed. And late binding when we want expressiveness and rapid time to market. Electronic trading systems are a perfect example of a class of problem where we want all those contradictory virtues in the same system. A combination of languages is the only way to satisfy all those requirements. Brad Cox laid all this out this approach with great prescience in Planning the Software Industrial Revolution in 1990. What we need a development environments that allows us to mix languages at will. Debuggers that cope seamlessly when we examine a stack with different languages in different frames. Consistent threading models across runtimes. And consistent memory management models. Microsoft have come closest to achieving this with the .Net CLR. The Java VM can host multiple languages too, but not with the same quality tooling.

Personally I like the combination of C++ and Python. But there are tensions implicit in building systems with that mix. The first is the threading model. If you’re doing server engineering you mus be GIL aware. A single C++ thread works well with Python. A limited number of C++ threads on dedicated tasks, with just one of them invoking Python works well too. Multiple C++ threads as pooled workers using locking to cooperate in executing identical logic and each invoking Python will not work well.

Another tension is the different memory management models. The Python C runtime organises it’s own memory pool of PyObject* allocations. There are separate sub pools for different types, with different rules for the management of strings, integers and larger objects. Python’s memory pool tends to only grow, unlike a C++ program who’s memory profile we can see fall when the C RT lib hands memory back to the OS.

So if we have multiple languages in the same run time one of the biggest challenges is making the right architectural decisions so that those languages cooperate despite drawing on the same OS provided resources in different ways.

mod_wsgi on Symbiosis

etrading — Sun, 08 Jan 2012 15:29:45 +0000

Bytemark’s Symbiosis Linux doesn’t have mod_wsgi installed in its Apache2 deployment. To make it work with the Python 2.7 I built from source I had to build mod_wsgi from source too. mod_wsgi’s Makefile needs Apache’s apxs2 tool, which isn’t in the standard Symbiosis. Fortunately that is in the apache-threaded-dev package. Here are the steps I used to make mod_wsgi work…

# get apxs2 so we can install mod-wsgi from source. Needs to be built from source to
# get the Python.h from the Python we built from source.
sudo apt-get install apache2-threaded-dev
# for mod_wsgi: configure should find apxs2
# the soft link is needed so that libtool can find gcc
./configure
 ln -s /usr/bin/gcc-4.3 /usr/bin/i486-linux-gnu-gcc
 make
 make install
 /etc/init.d/apache2 restart

Satchmo on Symbiosis

etrading — Tue, 20 Dec 2011 10:07:14 +0000

I’m going a little bit off topic here, as this post has no trading or finance content. However, it’s good to share, and hopefully this will help anyone out there struggling with a Django based web dev stack on a Bytemark Symbiosis host. I’m using Satchmo for an ecommerce site. But a lot of the detail will be applicable to any Django based web site. If you’re hosting plain old HTML, or PHP, then Symbiosis is fine and dandy. If you want to get Satchmo or Django running you’ve got a bit more work to do as Symbiosis is Bytemark’s own Debian Lenny distro with Python 2.5 and no GCC. The ecommerce site I’m deploying has been developed with Python 2.7.2, Django 1.3.1 and Satchmo 0.9.2. So I needed to build up the whole thing from scratch, starting with an apt-get for GCC and a Python 2.7.2 source build. Here’s the recipe I followed. Note that I’m omitting all the directory tree specific cds, gunzips and ‘tar xvf’s from the commands, but leaving in all the fiddly cmd line options that can take ages to figure out…

## GCC + Python 1.7
apt-get install gcc-4.3
wget www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
# We need to bld Python with zlib support compiled in, otherwise
# Satchmo's easy_install setuptools with fail with a zlib ImportError.
# Check that /usr/include/zlib.h exists
apt-get install zlib1g-dev
# We need the mercurial hg client to pull stuff off bitbucket repositories
apt-get install mercurial
# Also get pip installer
wget http://python-distribute.org/distribute_setup.py
python distribute_setup.py
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python get-pip.py
# Now build Python
./configure -with-zlib=/usr/include
make
make install
# Python 2.7.2 is now the default on this host
## sqlite 2.6.3 - not a std part of Linux source build
apt-get install libsqlite3-dev
wget pysqlite.googlecode.com/files/pysqlite-2.6.3.tar.gz
python setup.py install
## Django
wget https://www.djangoproject.com/download/1.3.1/tarball
python setup.py install
python
>>> import django
## Satchmo
wget https://bitbucket.org/chris1610/satchmo/get/v0.9.1.tar.gz
# gunzip, untar then follow Satchmo install notes. First, get easy_install. This will fail if
# you didn't get the zlib stuff right.
wget pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg
easy_install pycrypto
easy_install http://www.satchmoproject.com/snapshots/trml2pdf-1.2.tar.gz
easy_install django-registration
wget effbot.org/downloads/Imaging-1.1.7.tar.gz
python setup.py install
# ReportLab
wget www.reportlab.com/ftp/reportlab-2.5.tar.gz
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-threaded-multihost/
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-caching-app-plugins/
python setup.py install
pip install sorl-thumbnail==3.2.5
hg clone http://bitbucket.org/bkroeze/django-signals-ahoy/
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-livesettings/
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-keyedcache/
python setup.py install
hg clone http://bitbucket.org/chris1610/satchmo/
python setup.py install
python
>>> import django
>>> import satchmo_store
# Now clone the default Satchmo store
python scripts/clonesatchmo.py
python manage.py runserver
# Fix Symbiosis firewall to allow incoming on port 8000
# http://symbiosis.bytemark.co.uk/docs/ch-reference-firewall.html
cd /etc/symbiosis/firewall/incoming-d
touch 11-8000
firewall
# Finally, run satchmo. Note 0.0.0.0 to be available to any connection
python manage.py runserver 0.0.0.0:8000

Tales on Python: in praise of the GIL

etrading — Mon, 28 Nov 2011 14:23:02 +0000

Tales notes the trend towards Python in a recent post. Couple of points in response: firstly, slang at Goldman. I’m told by ex Goldmanites that slang is very close to Python anyway. Secondly, Python’s Global Interpreter Lock. The GIL always seems a disadvantage to developers accustomed to ‘traditional’ multi-threaded environments like C++, Java and C#. Those who criticise Python from that perspective are right to point out that the GIL prevents developers from achieving concurrency within a single process, because it puts locking around access to all memory owned by the Python runtime. That is a real constraint, but one that has benefits when dealing with mixed language environments. For example, if you’re implementing a pub sub API in C++ using Python’s C API, you’ll want a callback mechanism to handle incoming messages. Native pub sub APIs typically spawn another thread on which they dispatch new message callbacks. When that thread invokes the callback implementation, the callback implementation can invoke Python code without worrying about locking or mutexes or the fact that it’s invoking Python’s C API on a different thread than the main app thread. It’s not a concern, because the GIL serializes all access to Python’s memory. Which simplifies things hugely…

C++ is back ?

etrading — Wed, 14 Sep 2011 13:47:39 +0000

Tales blogs that C++ is back, and attributes it’s renewed significance to C++11 and Objective-C. I’d say it never went away. Of course C++ isn’t the right language for DB backed web form apps. But it’s absolutely the right choice for operating systems, drivers, gaming engines, pricing algorithms, HFT systems. Anything where efficiency, control and working closely with the hardware matter. I don’t see that changing anytime soon, since platform and language vendors are targeting the vast middle ground of “enterprise apps”. For “enterprise”, read boring…

Quantivity on Fund Structure Arbitrage

etrading — Mon, 29 Aug 2011 09:12:37 +0000

Fascinating post from Quantivity – I’m hoping for more on the same topic from him. Many of the advantages listed would be enjoyed by any small non real money fund: hedge, prop, family office etc. Of course there are some serious obstacles that small (relatively) unregulated funds face, and Lars Kroijer describes them in detail in Money Mavericks. And a lack of legacy technology is indeed an advantage in building trading systems quickly. A relatively recent pre existing framework, either from vendor or in house built can be a big advantage though. A classic example is gateways for exchange/ECN connectivity.

C++ is faster…

etrading — Sun, 12 Jun 2011 16:03:54 +0000

…Google says so here.

Model M keyboard

etrading — Thu, 07 Apr 2011 08:27:32 +0000

One of the lads at work has passed his classic Model M keyboard on to me. He’s got a more recently built version of the same thing. It’s a real retro computing experience. The solidity and tactile feedback from the keyboard is fantastic. And it takes me back to early coding days in the mid 80s on IBM PC AT kit…

getattr considered harmful

etrading — Tue, 08 Mar 2011 17:24:15 +0000

Today’s coding reminded me why avoid __getattr__ in Python. FYI, a class that implements a __getattr__ method overrides the member access operator “.” So simply referencing object.member invokes object.__getattr__( member). A typical debugging technique in Python is to litter code with print object.member. Guess what happens if you use that debugging technique inside a __getattr__ method ? Like, duh ?!

Lock free: the new C++ frontier

etrading — Wed, 09 Feb 2011 15:57:49 +0000

One of the joys of C++ is the way the state of the art in the language periodically renews itself. I started coding in C++ in 1992 (!?!). I spent a couple of years away in 2001 & 2, and when I came back to C++ in 2003 the generic programming movement led by the likes of Stepanov and Alexandrescu had renewed the language. _1 indeed ! And now it feels like it’s happening again. LMAX put me onto the lock free meme a couple of weeks back. There’s a Herb Sutter article on lock free queues out there, folks on stackoverflow say it’s difficult, Hack the Market has blogged about FastFlow, and there’s a “Boost” lock free lib.