Develop Excel: London 2018-10-18

SpreadServe will be at the Develop Excel conference in London on 18th October. If you build spreadsheets for the trading floor with XLLs you should be there too. I’m especially excited about the session with Govert, Mr Excel-DNA.

TFWebSock: TransFICC web socket server

I’ve just updated my Netty based TransFICC web socket server to work with version 330cb31-3670 of the TransFICC API and service. TFWebSock enables web socket clients to subscribe to TransFICC sourced market data. Those web socket clients could of course be browsers, but in the case of this demo the client is the SSAddin Excel addin. Which means that you can use TFWebSock and SSAddin to pull live ticking ICAP iSwap swap rates into Excel on your desktop. And if you want to automate and serverize that spreadsheet you can do so with SpreadServe.

I should point out that the ticking rates in this demo are from the iSwap test system, and are not live production rates. Many thanks to the teams at TransFICC and ICAP for making this data available!

Automated Baremetrics Spreadsheet

Back in April I read a great post by Jaakko Piipponen on the SaaS financial model he’d built as an Excel spreadsheet. I’m interested in spreadsheet financial models, so I enjoyed Jaakko’s run down on how his model covers PnL report, operating expenses, payroll and revenue. Jaakko describes how you only need to spend 30 mins a month downloading reports from the Baremetrics dashboard and pasting it into Excel to keep it up to date. The post is titled an “SaaS financial model you’ll actually update”, because the value of the model is such that 30 mins of manual handle turning a month is more than justified. Now I’m the first to advocate Excel for business agility; it’s a great tool that enables sales, marketing and finance people to solve problems for themselves without the bottleneck of a software development team. But the downside of Excel is manual operation like the download and paste Jaakko spells out. Wouldn’t it be great if the Baremetrics data in the spreadsheet downloaded and updated automatically? Then the model would always be up to date, and no tedious and error prone download’n’paste operations are necessary.

So I built it by adding Baremetrics functions to my open source XLL Excel Addin SSAddin. Here’s a spreadsheet that strips Jaako’s model down to just the revenue parts of the model…

https://github.com/SpreadServe/SSAddin/blob/master/xls/flight_bare_model2.xls

It uses SSAddin functions to automatically download the revenue numbers every 10 minutes into the MRR Export sheet via the Show Summary API so the Revenue Model sheet will update every 10 minutes with the latest new customers, subscriptions, cancellations, upgrades and downgrades. All you need to do is download SSAddin 32 or 64 bit .xll binaries from here…

http://spreadserve.com/s3/downloads.html

Then install the addin in Excel and fire up the spreadsheet. Don’t forget to ensure that auto calc is on, and that you’ve got a copy of SSAddin.xll.config from the download page and put it next to SSAddin.xll on your PC. You’ll need to edit the .config to add your Baremetrics license key.

You can see flight_bare_model2.xls running online and driven by spreadserve.com data here…

http://sscalc0.online/flight_bare_model2.xls/0

http://sscalc0.online/flight_bare_model2.xls/MRR%20Export

This online spreadsheet takes automation to the ultimate level. When you run a spreadsheet model automated with SSAddin on your desktop or laptop you still have to start Excel, load the spreadsheet and hit F9 before auto updates can start. With SpreadServe hosting, you can move the spreadsheet onto a cloud server. All manual operations are eliminated, and everyone sees the same auto updating numbers in their browsers.

SSAddin: Google Analytics, Baremetrics, Quandl & Tiingo

I’ve mentioned SSAddin on this blog before, but not given much detail about it. SSAddin is an XLL Excel addin exposing APIs for Google Analytics, Baremetrics, Quandl, Tiingo as worksheet functions. It’s distributed under a permissive Apache 2.0 license, so you can use, repurpose or redistribute however you like with no fees so long as you include license and copyright notices. You can download 32 and 64 bit .xll binaries from here: http://spreadserve.com/s3/downloads.html

Don’t forget to take a copy of SSAddin.xll.config too and put it next to SSAddin.xll on your PC. You’ll need to edit the .config to add your license keys for Google, Baremetrics, Quandl and Tiingo. There’s documentation here, but I suggest you start by playing with example spreadsheets from GitHub: https://github.com/SpreadServe/SSAddin/tree/master/xls

Don’t hesitate to ask questions in the comments here, or on the download page, or to raise an issue on GitHub.

 

SpreadServe AMI part III

The SpreadServe 0.4.2b AMI is now available in the US West Oregon region: just search for SpreadServe in public images. There’s a now YouTube video on launching a SpreadServe AMI.

Two points of interest came up in preparing the AMI: Admin password persistence, and the region bound nature of AMIs. This blog from 2013 suggests that Admin passwords don’t persist in AMIs, but I’ve found they do now. So the Administrator password for a SpreadServe 0.4.2b AMI is SpreadServe042b. If you launch your own SpreadServe instance using the AMI, then I suggest you change the password!

The SpreadServe 0.4.2b AMI is only published in the US West Oregon region. Only AMI owners can copy to another region, so if you want a SpreadServe AMI in another region you’ll need to do a simple workaround: start an image in US West Oregon, stop it and create your own image, then copy to your preferred region.

IEX market data, CreateEvent & CPU spikes

This week I’ve been testing the SpreadServe addin with Tiingo’s IEX market data. I was checking performance on my sscalc0.online AWS host for a group of SpreadServeEngines executing various test and demo spreadsheets, including one that subscribes to IEX tickers for AAPL & SPY, via Tiingo API websockets. That API gives us real time top of book as well as last trade price and size for the cash equity traded on IEX. In my test scenario I was running five engines, two of them idle, three running spreadsheets, one of which was a simple IEX market data subscriber. Using Process Explorer I saw some odd CPU spiking on the idle engines. Zooming in with Process Explorer I could see the busyness was on a thread that should have been idle, sleeping inside a WaitForSingleObject call, waiting for a signal to check its input queue. The event object waited upon was created by some generic code invoking win32’s CreateEvent and also used in another thread. Reading the docs I found that CreateEvent’s fourth param, the event object name, implies that the caller will get a handle to a previously created event object if the names match. And I was using a hardwired name! So my thread was being repeatedly woken by events from another thread. A quick fix to make the names unique produced idling engines with no unnecessary CPU burn. All very instructive, partly because running on AWS makes one very aware of paying by the CPU hour.

SpreadServe AMI part II

The core component in a SpreadServe deployment is the SpreadServeEngine, a headless C++ server binary that implements the Excel compatible calculation engine. The engine discovers its hostname through the win32 API using GetComputerNameExA( ComputerNameDnsFullyQualified, …). On AWS this was giving me hostnames like WIN-THU4IQNRN6F, when what I wanted was the fully qualified domain name, like ec2-54-186-184-85.us-west-2.compute.amazonaws.com. Harry Johnston helpfully advised on StackExchange that, since the host is not joined to a domain, GetComputerNameExA will only return the FQDN if I explicitly set it via Control Panel. Naturally I want to avoid manual fixes on a SpreadServe AMI so I settled on using Amazon’s EC2 instance metadata. The FQDN hostname can be discovered with an HTTP GET on this URL from any EC2 host: http://169.254.169.254/latest/meta-data/public-hostname. I built a small helper server process to query instance metadata using Tornado’s async HTTPClient and write it to the localFS, where SpreadServeEngine can read it. Result: any new SpreadServe AMI will automatically discover its public DNS.

Threading is hard: Tiingo, IEX & SSAddin

Recently I’ve been rediscovering the fact that threading is hard. I’ve been extending the SpreadServe Addin to support Tiingo‘s IEX market data feed. Real live ticking market data is usually only found inside investment banks, brokers and big hedge funds as it takes a lot of cash and infrastructure to connect to exchanges directly or to subscribe via Reuters. Even newer internet contenders like xignite are very expensive too. Tiingo’s IEX feed provides live ticking equity top of book data at an unprecedented price point. That is an exciting new development that I want to support in SSAddin. Coding it up has renewed my appreciation of how tricky multithreaded code can be. The SSAddin is implemented in C# packaged as an XLL using ExcelDNA. As with any Excel XLL, the worksheet functions it defines are executed on the main Excel thread. If they are long running, then they’ll block the GUI. So the worksheet functions pass off their work to a background thread. This means that SSAddin can do quandl and tiingo historical data queries without blocking the main Excel thread. Query results are cached, and there’s a set of worksheet functions to pull results out of the cache. So far so good. However, adding subscriptions to Tiingo’s IEX market data adds more complexity. In .net callbacks for web socket events are dispatched on pool threads. Ticking data is pushed back into Excel via RTD. So lots of lock statements are necessary to coordinate access to the queue for passing work from the Excel thread to the background thread, and for coordinating access to subscription management data structures and the RTDServer between the background thread and the pool threads that dispatch the socket callbacks. All good fun which has prompted a few thoughts. Firstly, threading is hard! Secondly, I must get round to learning Rust and understanding the borrow checker. Thirdly, thanks heavens for lock reentrancy in .net!

Spreadsheets are code

Felienne Hermans has made it her mission to point out that “spreadsheets are code”. She’s most definitely right about that, and a whole host of the other consequences that she draws from that insight, specifically that we should apply the techniques developed by mainstream software engineering to spreadsheets: version control, testing and design guidelines for clean structure, like the FAST standard. Whenever you create a sheet with formulae in it you’re programming. Ignoring that fact is one of the reasons spreadsheet disasters keep happening. I couldn’t agree more with Prof Hermans on that score. But I think we need to go further in the comparison of spreadsheets with code, and point out some major differences.

1. Conventional code, when deployed to its production runtime environment does not come with an IDE that enables any user to change the implementation! A trader can’t reach inside his Bloomberg or TradeWeb terminal and change its implementation. But Excel allows any user to change any formula in a financial model.
2. Well structured conventional code separates user interface from business logic. This separation of concerns is often called the MVC pattern. A typical modern web application has an HTML/CSS/JavaScript user interfaces running in browsers talking to a server hosted backend coded in Python on top of Django and an RDBMS. Excel makes it impossible to decouple the user interface from the logic expressed in formulae and VBA. Modern applications present a choice for their architects; does a given piece of code belong in the browser, the server tier or the DB ?
3. Conventional code enables reuse through components. Each Excel spreadsheet is like an island, and monolithic. How can spreadsheets be composed together to draw input and feed output to each other? Only with manual, error prone operations.
4. Unit testing: the unit testing philosophy calls for any significant component to have a set of separate test code that proves compliance with pre and post conditions as well as yielding specified results. Also required is the ability to run a set of tests automatically and record the results. All of that is a capability that Excel simply doesn’t have.

To realise points 1 to 4 for spreadsheets we need an alternate run time that can host spreadsheets on a server, and decouple the financial logic expressed in worksheet formulae, VBA & XLLs from the user interface. In the next post I’ll give more detail on how SpreadServe solves all the issues raised above.