mod_wsgi on Symbiosis

Bytemark’s Symbiosis Linux doesn’t have mod_wsgi installed in its Apache2 deployment. To make it work with the Python 2.7 I built from source I  had to build mod_wsgi from source too. mod_wsgi’s Makefile needs Apache’s apxs2 tool, which isn’t in the standard Symbiosis. Fortunately that is in the apache-threaded-dev package. Here are the steps I used to make mod_wsgi work…

# get apxs2 so we can install mod-wsgi from source. Needs to be built from source to
# get the Python.h from the Python we built from source.
sudo apt-get install apache2-threaded-dev
# for mod_wsgi: configure should find apxs2
# the soft link is needed so that libtool can find gcc
./configure
 ln -s /usr/bin/gcc-4.3 /usr/bin/i486-linux-gnu-gcc
 make
 make install
 /etc/init.d/apache2 restart

 

Satchmo on Symbiosis

I’m going a little bit off topic here, as this post has no trading or finance content. However, it’s good to share, and hopefully this will help anyone out there struggling with a Django based web dev stack on a Bytemark Symbiosis host. I’m using Satchmo for an ecommerce site. But a lot of the detail will be applicable to any Django based web site. If you’re hosting plain old HTML, or PHP, then Symbiosis is fine and dandy. If you want to get Satchmo or Django running you’ve got a bit more work to do as Symbiosis is Bytemark’s own Debian Lenny distro with Python 2.5 and no GCC. The ecommerce site I’m deploying has been developed with Python 2.7.2, Django 1.3.1 and Satchmo 0.9.2. So I needed to build up the whole thing from scratch, starting with an apt-get for GCC and a Python 2.7.2 source build. Here’s the recipe I followed. Note that I’m omitting all the directory tree specific cds, gunzips and ‘tar xvf’s from the commands, but leaving in all the fiddly cmd line options that can take ages to figure out…

## GCC + Python 1.7
apt-get install gcc-4.3
wget www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
# We need to bld Python with zlib support compiled in, otherwise
# Satchmo's easy_install setuptools with fail with a zlib ImportError.
# Check that /usr/include/zlib.h exists
apt-get install zlib1g-dev
# We need the mercurial hg client to pull stuff off bitbucket repositories
apt-get install mercurial
# Also get pip installer
wget http://python-distribute.org/distribute_setup.py
python distribute_setup.py
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python get-pip.py
# Now build Python
./configure -with-zlib=/usr/include
make
make install
# Python 2.7.2 is now the default on this host
## sqlite 2.6.3 - not a std part of Linux source build
apt-get install libsqlite3-dev
wget pysqlite.googlecode.com/files/pysqlite-2.6.3.tar.gz
python setup.py install
## Django
wget https://www.djangoproject.com/download/1.3.1/tarball
python setup.py install
python
>>> import django
## Satchmo
wget https://bitbucket.org/chris1610/satchmo/get/v0.9.1.tar.gz
# gunzip, untar then follow Satchmo install notes. First, get easy_install. This will fail if
# you didn't get the zlib stuff right.
wget pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg
easy_install pycrypto
easy_install http://www.satchmoproject.com/snapshots/trml2pdf-1.2.tar.gz
easy_install django-registration
wget effbot.org/downloads/Imaging-1.1.7.tar.gz
python setup.py install
# ReportLab
wget www.reportlab.com/ftp/reportlab-2.5.tar.gz
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-threaded-multihost/
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-caching-app-plugins/
python setup.py install
pip install sorl-thumbnail==3.2.5
hg clone http://bitbucket.org/bkroeze/django-signals-ahoy/
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-livesettings/
python setup.py install
hg clone http://bitbucket.org/bkroeze/django-keyedcache/
python setup.py install
hg clone http://bitbucket.org/chris1610/satchmo/
python setup.py install
python
>>> import django
>>> import satchmo_store
# Now clone the default Satchmo store
python scripts/clonesatchmo.py
python manage.py runserver
# Fix Symbiosis firewall to allow incoming on port 8000
# http://symbiosis.bytemark.co.uk/docs/ch-reference-firewall.html
cd /etc/symbiosis/firewall/incoming-d
touch 11-8000
firewall
# Finally, run satchmo. Note 0.0.0.0 to be available to any connection
python manage.py runserver 0.0.0.0:8000

Tales on Python: in praise of the GIL

Tales notes the trend towards Python in a recent post. Couple of points in response: firstly, slang at Goldman. I’m told by ex Goldmanites that slang is very close to Python anyway. Secondly, Python’s Global Interpreter Lock. The GIL always seems a disadvantage to developers accustomed to ‘traditional’ multi-threaded environments like C++, Java and C#. Those who criticise Python from that perspective are right to point out that the GIL prevents developers from achieving concurrency within a single process, because it puts locking around access to all memory owned by the Python runtime. That is a real constraint, but one that has benefits when dealing with mixed language environments. For example, if you’re implementing a pub sub API in C++ using Python’s C API, you’ll want a callback mechanism to handle incoming messages. Native pub sub APIs typically spawn another thread on which they dispatch new message callbacks. When that thread invokes the callback implementation, the callback implementation can invoke Python code without worrying about locking or mutexes or the fact that it’s invoking Python’s C API on a different thread than the main app thread. It’s not a concern, because the GIL serializes all access to Python’s memory. Which simplifies things hugely…

C++ is back ?

Tales blogs that C++ is back, and attributes it’s renewed significance to C++11 and Objective-C. I’d say it never went away. Of course C++ isn’t the right language for DB backed web form apps. But it’s absolutely the right choice for operating systems, drivers, gaming engines, pricing algorithms, HFT systems. Anything where efficiency, control and working closely with the hardware matter. I don’t see that changing anytime soon, since platform and language vendors are targeting the vast middle ground of “enterprise apps”. For “enterprise”, read boring…

Quantivity on Fund Structure Arbitrage

Fascinating post from Quantivity – I’m hoping for more on the same topic from him. Many of the advantages listed would be enjoyed by any small non real money fund: hedge, prop, family office etc. Of course there are some serious obstacles that small (relatively) unregulated funds face, and Lars Kroijer describes them in detail in Money Mavericks. And a lack of legacy technology is indeed an advantage in building trading systems quickly. A relatively recent pre existing framework, either from vendor or in house built can be a big advantage though. A classic example is gateways for exchange/ECN connectivity.

C++ is faster…

…Google says so here.

Model M keyboard

One of the lads at work has passed his classic Model M keyboard on to me. He’s got a more recently built version of the same thing. It’s a real retro computing experience. The solidity and tactile feedback from the keyboard is fantastic. And it takes me back to early coding days in the mid 80s on IBM PC AT kit…

__getattr__ considered harmful

Today’s coding reminded me why avoid __getattr__ in Python. FYI, a class that implements a __getattr__ method overrides the member access operator “.”  So simply referencing object.member invokes object.__getattr__( member). A typical debugging technique in Python is to litter code with print object.member. Guess what happens if you use that debugging technique inside a __getattr__ method ?  Like, duh ?! 

Lock free: the new C++ frontier

One of the joys of C++ is the way the state of the art in the language periodically renews itself. I started coding in C++ in 1992 (!?!). I spent a couple of years away in 2001 & 2, and when I came back to C++ in 2003 the generic programming movement led by the likes of Stepanov and Alexandrescu had renewed the language. _1 indeed ! And now it feels like it’s happening again. LMAX put me onto the lock free meme a couple of weeks back. There’s a Herb Sutter article on lock free queues out there, folks on stackoverflow say it’s difficult, Hack the Market has blogged about FastFlow, and there’s a “Boost” lock free lib.

Limit Order Book implementation

Fascinating blog on HFT implementation from WK. He commnts “a variation on this structure is to store the Limits in a sparse array instead of a tree. “  More detail on the implications for L1 and L2 cache behaviour of trees versus arrays for limits would be welcome. I’m assuming C++ implementation here of course, though WK points out you can make Java go fast if you avoid GC, which chimes with the experience of the LMAX guys. I ask because I interviewed with a big HFT firm last year: they gave me a programming exercise based on L1/2 cache behaviour.