March 13, 2017
Recently I’ve been using the Enron spreadsheets to test SpreadServe, simply because it’s good testing practice to expose any codebase to a high volume of diverse inputs. Felienne made them available on figshare, but they’re in a slightly obscure zip format, so I posted them on github to make them more accessible. SpreadServe posts information about the formulae used in each sheet into the spreadserve.com DB, so I did a simple analysis of Enron formula use. The results are on github here. To summarise: there are 15927 sheets, and only 8421 use formulae. 152 different functions are used across all sheets, and only 170 sheets use maths funcs that go beyond arithmetic. So the Enron spreadsheets weren’t as diverse as I’d hoped. They made for a good volume test though. Here’s a short video about exploring the Enron spreadsheets with SpreadServe…
March 6, 2017
The SpreadServe 0.4.2b AMI is now available in the US West Oregon region: just search for SpreadServe in public images. There’s a now YouTube video on launching a SpreadServe AMI.
Two points of interest came up in preparing the AMI: Admin password persistence, and the region bound nature of AMIs. This blog from 2013 suggests that Admin passwords don’t persist in AMIs, but I’ve found they do now. So the Administrator password for a SpreadServe 0.4.2b AMI is SpreadServe042b. If you launch your own SpreadServe instance using the AMI, then I suggest you change the password!
The SpreadServe 0.4.2b AMI is only published in the US West Oregon region. Only AMI owners can copy to another region, so if you want a SpreadServe AMI in another region you’ll need to do a simple workaround: start an image in US West Oregon, stop it and create your own image, then copy to your preferred region.
February 23, 2017
This week I’ve been testing the SpreadServe addin with Tiingo’s IEX market data. I was checking performance on my sscalc0.online AWS host for a group of SpreadServeEngines executing various test and demo spreadsheets, including one that subscribes to IEX tickers for AAPL & SPY, via Tiingo API websockets. That API gives us real time top of book as well as last trade price and size for the cash equity traded on IEX. In my test scenario I was running five engines, two of them idle, three running spreadsheets, one of which was a simple IEX market data subscriber. Using Process Explorer I saw some odd CPU spiking on the idle engines. Zooming in with Process Explorer I could see the busyness was on a thread that should have been idle, sleeping inside a WaitForSingleObject call, waiting for a signal to check its input queue. The event object waited upon was created by some generic code invoking win32’s CreateEvent and also used in another thread. Reading the docs I found that CreateEvent’s fourth param, the event object name, implies that the caller will get a handle to a previously created event object if the names match. And I was using a hardwired name! So my thread was being repeatedly woken by events from another thread. A quick fix to make the names unique produced idling engines with no unnecessary CPU burn. All very instructive, partly because running on AWS makes one very aware of paying by the CPU hour.
January 16, 2017
Recently I’ve been working on building an EC2 AMI for SpreadServe, so deployment becomes a one click operation for Amazon AWS users. I ran into an interesting snag so I thought I’d capture it here. My aim was to deploy SpreadServe as a Windows Service on an AWS Windows Server 2012 R2 image, so I used pywin32‘s excellent win32service module. Here’s my github boilerplate project for a Windows Service in Python. On my AWS host my SpreadServe Windows Service was failing to start, and leaving no trace in the system or application event logs. pywin32service has a debug mode; when I tried that I got a Windows 0xc00007b error, which indicates a mix of 32 and 64 bit binaries. SpreadServe is 32 bit all the way, so something was wrong. I turned to procmon to try and figure what was failing. procmon showed that my 32 bit pythonservice.exe was loading a 64 bit python27.dll, instead of the 32 bit python27.dll that’s part of the SpreadServe install tree. The 64 bit DLL was coming from the C:\Program Files\Amazon\cfn-bootstrap directory, which is added to the standard Windows 2012 R2 image by Amazon to support CloudFormation, and is on the system path. After much experimenting I couldn’t find a way to stop Windows Service Host from using the system path, so I had to change it to replace cfn-bootstrap with SpreadServe directories. Problem solved…