Colin Lancaster’s Fed Up

Colin Lancaster’s Fed Up is a new addition to the literature on trading financial markets, promising the inside line on running a macro fund operating in the global markets during the Covid pandemic. Having worked as a sell side techie building trading systems for the last twenty years I’ve read a fair bit of this literature, from classics like Niederhoffer’s Education of a Speculator and Kindelberger’s Mania, Panics & Financial Crashes, to riotous shock fests like Belfort’s Wolf of Wall Street and Anderson’s Cityboy: Beer and Loathing in the Square Mile. For me the real standouts manage to combine market history, financial mechanics exegesis, storytelling and amusing anecdotage. The best example is Michael Lewis’ Liars Poker. Partnoy’s FIASCO and Das’ Traders, Guns and Money are commendable as well. So how does Lancaster’s effort stack up against the canon?

Lancaster plays it slightly differently as he presents a thinly novelized account of his experiences, rather than memoir or narratized tutorial. Because he presents his account as a novel I will mark him down on a couple of points later, but firstly I must recognize his major achievement with this book. Lancaster succeeds in his dramatic reconstruction of the flow of political and economic events week by week since Q1 2020. The lightening quick market reversals, central bank interventions, major political developments, and their impact on a fund manager and his team are described with gripping momentum, and rendered in such convincing detail that I could almost see the teeth grinding desk thumping traders hunched over their Bloombergs.

A novelized treatment gives Lancaster the scope to wander, so along the way we take in a lot of enjoyable pop cultural references. Amy Winehouse, Led Zep, the Stones, Guns’n’Roses, U2 and Freddie Mercury all get mentions. Even Dr John, and that’s one for the real musos! There’s even time for a Hunter S Thompson quote. Fiction also enables Lancaster to lecture his readers on pet economic and political topics. He uses the literary device of mentoring a junior to deliver set pieces on twentieth century fiscal policy, and the dislocation of quantitative easing as the “new normal”. Personally I found mentions of Clinton as the last President to balance the US budget, and the Reinhard-Rogoff paper on government debt to GDP ratios gratifying. Especially as the latter was used by Cameron and Osborne to justify austerity.

So where does Fed Up fall down? If you come looking for detailed explanation of the kind presented in Zuckerman’s Greatest Trade, you’ll be disappointed. If you don’t know what yield curve inversion is, Lancaster’s not going to enlighten you. The presentation of the story as fiction causes some irritation: there’s an Amy Winehouse dream sequence that serves no apparent purpose. And the relationship of the principal with his wife is referenced intermittently, with hints that his neglect has induced her to have an affair, but then it’s all left unresolved. There’s also a slightly sanctimonious tone taken on drug using colleagues that makes me suspect the author doth protest too much. And finally a couple of minor typos and factual errors: there’s no company called HIS Markit – it’s “IHS Markit”, and Lancaster is wrong about the timing of the death of Brian Jones. If he’s as big a music fan as he seems he should have got that one right. But I do appreciate his story about a team outing to the hotel bar where the Stones launched Beggars Banquet back in 68. That was new to me!

In conclusion, I think I’ll rate this as 3/5. Don’t pick this one up if you haven’t read Liars Poker. But if you are familiar with the classics you’ll find it thought provoking, entertaining and enjoyable.

Showstopper!

I’ve just finished reading Zachary’s Showstopper, a riveting account of the development of Windows NT at Microsoft in the late 80s and early 90s. I do love industry war stories, so I’m puzzled by the fact I missed this book somehow. Other similar works I’ve enjoyed include Ferguson’s No Prisoners, Kidder’s Soul of a New Machine and Levy’s Hackers. The book is slightly marred at times by clunky explanations of core development activities like coding, test, debugging and building. Zachary is a tech journo, but obviously not a coder. Beyond that minor objection there’s much to enjoy, and several surprises. Those who’ve been around the software world for the last couple of decades will recognize the dramatis personae: Bill Gates, Steve Ballmer, Nathan Myrhvold, and the NT lead, Dave Cutler, developer of VMS at DEC. The narrative follows an arc familiar to any who have worked on a large software project, with all the typical setbacks and pressures. Here are the points that surprised me…

• Cutler joined MS in Oct 88, and NT development was well underway before the decision to make NT the successor to Windows 3. Originally OS/2 had taken that role. But the unexpected success of Windows 3 in 1990, and Microsoft’s strained relationship with IBM changed all that. Before that NT was a traditional multitasking console OS like VMS or Unix.
• Cutler had three prime design directives: reliability, personality and portability. Reliability was achieved with a properly partitioned kernel mode, unlike DOS & Windows. Personality was accomplished by presenting a different user mode API (Win32) to apps, and keeping the kernel API private. Which much later in WinNT’s history would enable WSL version 1. Personality also enabled Windows 3 apps to run under NT. Portability meant minimizing assembly coding.
• Cutler original target CPU was Intel’s RISC offering, the i860. A CPU so new and buggy that Cutler brought his hardware buddy Roy Short over from DEC to build i860 based systems at MS. Gates wanted to target 80386 first, but deferred to Cutler on many key decisions. Later support for MIPS and DEC Alpha would be added. I remember seeing Alpha based NT workstations in the mid 90s and being very impressed by the power and speed.
• Manual builds! There was no CI/CD automation back then. The guys running the “Build Lab” selected the patches they would take, merged, and built. And the build took 19 hours.
• Most of NT was coded in C, but much of the graphics was in C++. The graphics coders were long time Microsofties, not Cutler cronies from DEC, and they had Gates support in C++. Since Windows graphics only entered the NT picture in 1990, and Cutler was not a GUI guy, they didn’t have to follow his language choice. I remember early C++ compilers – Glockenspiel CommonView anyone? Coding part of a new OS in C++ in 1990 was a very bold choice.
• 20% of the NT team were first jobbers!

Will we ever see a big new OS dev project again? Maybe not. Cutler’s Personality idea enabled the pivot to Windows by NT, but now we have virtualization and cheap hardware, so the Personality job is done by hypervisors. Just look at WSL2 compared to WSL1! OSes have become commoditized, and since Linux is open source, it enables a style of innovation that proprietary OSes can’t match. So, in conclusion, Showstopper is a strong recommend from me, and a nostalgic read for this veteran dev. Almost a “when dinosaurs roamed the earth” experience. And Cutler may have been difficult to work with, but he obviously had massive coding chops.

Develop Excel: London 2018-10-18

SpreadServe will be at the Develop Excel conference in London on 18th October. If you build spreadsheets for the trading floor with XLLs you should be there too. I’m especially excited about the session with Govert, Mr Excel-DNA.

Windows EOL

Could Windows die as an enterprise OS? Could it disappear in the same way as desktop  workstations from Sun, SGI & Next? Could it vanish from the datacentre just like Solaris?

Twenty years ago Open Source advocates pushed the Linux desktop as a means to break Microsoft’s stranglehold on PC computing. At the same time Sun promoted Java applets as desktop OS agnostic. In reality it was the iPhone launch in 2007, and the rise of the smart phone that broke Microsoft’s dominance in the consumer space; Android & iOS are the leading phone and tablet OSes and have driven the shift from Windows being the primary GUI to HTML5. But despite the primacy of the HTML5 GUI for new systems, Windows position on the corporate desktop still seems secure. Why?

There are good reasons Windows persists in the business environment when it has receded in the consumer space: corporate knowledge workers need desktop computing, and Windows is still the best option: cheaper than MacOS, easier to use than Linux and more compatible with legacy apps than either. It’s still the best platform for running MS Office, naturally. But maybe ReactOS – an Open Source Windows clone, could eventually change that. With no license fee that makes it cheaper than Windows, so the question becomes: could ReactOS form the basis for a RedHat style play that would eliminate desktop Windows in the same way RedHat Enterprise Linux finished off Solaris? For that to be happen possible ReactOS would need to achieve higher levels of compatibility. Currently it can run a lot of apps including Excel Viewer, but not Office itself. That could be addressed by industry consortia. How much do large enterprises spend on Windows licensing? Potentially, that cost could be taken out in favour of funding consortium efforts to raise ReactOS compatibility, package it for corporates and supply support.

It’s an intruiging prospect, but there are good reasons why it would be hard to execute. Firstly, the sheer technical difficulty of troubleshooting without source code access for MS Office, .Net and other legacy apps. Debugging is a hell of a lot easier with source code access. Secondly, Microsoft’s competitive response; they could drop the cost of Windows volume licensing to zero or near zero. And they could engineer Office and .Net for ReactOS incompatibility. However, both responses could invoke resistance from anti monopoly regulators.

SpreadServe AMI part II

The core component in a SpreadServe deployment is the SpreadServeEngine, a headless C++ server binary that implements the Excel compatible calculation engine. The engine discovers its hostname through the win32 API using GetComputerNameExA( ComputerNameDnsFullyQualified, …). On AWS this was giving me hostnames like WIN-THU4IQNRN6F, when what I wanted was the fully qualified domain name, like ec2-54-186-184-85.us-west-2.compute.amazonaws.com. Harry Johnston helpfully advised on StackExchange that, since the host is not joined to a domain, GetComputerNameExA will only return the FQDN if I explicitly set it via Control Panel. Naturally I want to avoid manual fixes on a SpreadServe AMI so I settled on using Amazon’s EC2 instance metadata. The FQDN hostname can be discovered with an HTTP GET on this URL from any EC2 host: http://169.254.169.254/latest/meta-data/public-hostname. I built a small helper server process to query instance metadata using Tornado’s async HTTPClient and write it to the localFS, where SpreadServeEngine can read it. Result: any new SpreadServe AMI will automatically discover its public DNS.

RethinkDB batch insert performance

The forthcoming cloud version of SpreadServe uses a Tornado based server to persist a breakdown of all formulae used in a spreadsheet loaded by SpreadServe. For complex sheets I found that the insertion of many formulae in the formula table could be timeconsuming. In one test scenarion a multi-formula insert took 5 minutes. So I checked out the RethinkDB’s troubleshooting page where there are some useful performance tips. Batch insertions with the recommended batch size of 200 brought the insert time down from 5 mins to 21 secs. Further improvements came from using soft durability and noreply, bringing the insert time down to ~3.5 secs. However, I found that my Tornado server couldn’t respond to incoming HTTP GETs while the insert coroutine was looping on the insert batches. I figured that noreply meant that the yield in the loop resumed immediately, without waiting for the reply IO from the DB. Taking out noreply allowed the single threaded server to handle HTTP GETs in the middle of an insert. If improved performance is necessary in future, splitting the Tornado server into two processes may be the way to go, but for current test scenarios performance is acceptable.

Threading is hard: Tiingo, IEX & SSAddin

Recently I’ve been rediscovering the fact that threading is hard. I’ve been extending the SpreadServe Addin to support Tiingo‘s IEX market data feed. Real live ticking market data is usually only found inside investment banks, brokers and big hedge funds as it takes a lot of cash and infrastructure to connect to exchanges directly or to subscribe via Reuters. Even newer internet contenders like xignite are very expensive too. Tiingo’s IEX feed provides live ticking equity top of book data at an unprecedented price point. That is an exciting new development that I want to support in SSAddin. Coding it up has renewed my appreciation of how tricky multithreaded code can be. The SSAddin is implemented in C# packaged as an XLL using ExcelDNA. As with any Excel XLL, the worksheet functions it defines are executed on the main Excel thread. If they are long running, then they’ll block the GUI. So the worksheet functions pass off their work to a background thread. This means that SSAddin can do quandl and tiingo historical data queries without blocking the main Excel thread. Query results are cached, and there’s a set of worksheet functions to pull results out of the cache. So far so good. However, adding subscriptions to Tiingo’s IEX market data adds more complexity. In .net callbacks for web socket events are dispatched on pool threads. Ticking data is pushed back into Excel via RTD. So lots of lock statements are necessary to coordinate access to the queue for passing work from the Excel thread to the background thread, and for coordinating access to subscription management data structures and the RTDServer between the background thread and the pool threads that dispatch the socket callbacks. All good fun which has prompted a few thoughts. Firstly, threading is hard! Secondly, I must get round to learning Rust and understanding the borrow checker. Thirdly, thanks heavens for lock reentrancy in .net!

Windows Containers

Recently Microsoft has added support for containers to Windows Server. It’s available on the Azure cloud on VMs running Windows Server 2016 Tech Preview 3. I’ve been playing with it and I’ve got SpreadServe running inside a container. There’s much more detail here. But to summarise I found three workarounds were necessary…

• A two step process to build images as Windows container doesn’t like SpreadServe’s NSIS installer
• Web server inside the container should be on port 80 only internally
• A one line launch script that sets up environment variables is necessary

Very Sleepy performance profiling

Recently I’ve been using the excellent Very Sleepy profiler to performance tune the SpreadServeEngine’s loader, compiler and interpreter. One of our beta users had helpfully supplied a very large spreadsheet which was causing very long load and calc cycles. Back in the 90s you had to spend serious money on licenses for Pure Software‘s Purify and Quantify tools for this kind of work. Now tools like Dr Memory and Very Sleepy are free and OSS. It’s a while since I did a real, systematic performance profiling and tuning exercise, and I was soon reminded of how quickly preconceptions about which parts of the code might be CPU hogs can be shattered. It wasn’t long before I was soon nose to nose with one of the eternal truths of C++ development, or indeed development in any language: malloc & free are expensive. That’s why the LMAX team coded their own Java Collections. Printf isn’t cheap either. The answer was to introduce memory pooling for many of the most heavily used compiler and interpreter classes, and to set up config switches for the interpreter tracing. Interpreter tracing needs to be available in release builds as it a tremendously useful way of looking inside the execution of your spreadsheet. The result was a thirty fold improvement in load time, and much snappier calc cycles on very large sheets.

Upgrading the SpreadServe GUI

Recently I’ve been updating SpreadServe‘s JavaScript web GUI. The edit facility presented on the livesheet page was quite crude. It used a bootstrap-editable to pop up an edit field in a message box. Not really a seamless user experience, so I’ve upgraded to a JavaScript grid that supports in place editing. The end result is a user experience closer to Excel itself. I chose to use webismymind’s editablegrid since it can attach to an existing HTML table. In the case of SpreadServe the HTML table is generated in C++ code inside the SpreadServeEngine whenever a running spreadsheet is published to the RealTimeWebServer. I changed the C++ code to add attributes to the table and tr elements that editablegrid uses to latch on to. I made a couple of small changes to the editablegrid JavaScript so that the isEditable and modelChanged callbacks passed through more information. My isEditable implementation needed to get hold of the DOM element as well as the row column address so it could check for an ssedit attribute. And modelChanged needed the element ID so it could POST the changed data back to the RealTimeWebServer with a unique ID. The RealTimeWebServer sends the updated value back in to the SpreadServeEngine. I’ve forked editablegrid on github here with my changes. While I was doing this JS dev I found two resources very helpful: Oscar Rotero’s jQuery cheat sheet, and the Chrome debugger. I was a Firefox advocate for a long time, as Firebug was the first JavaScript debugger I used. The Chrome debugger is just faster, snappier, breakpointing is easier, and examining the DOM and variables seems better too.