Suppose you have a third party Win32 DLL, but don’t have access to the source code. The DLL exports a well known init function that makes callbacks into your code to register more functions. So you can call Win32’s LoadLibrary to load up the DLL, and you can use GetProcAddress to get hold of a pointer to the init func. You can invoke the init function from your code because you know the function prototype – the number and type of parameters. Then you get callbacks into a Register function you supply, which gives your code the names of other functions exported by the third party DLLs, as well as the number and type of parameters. Excel developers will recognise the description of the XLL function registration mechanism. So, given the names of those functions you can use GetProcAddress to get function pointers for them. But how do you invoke them? You don’t have function declarations available at compile time in your code. The functions don’t use varargs, so you can’t use va_start and va_end to figure out the params at run time.

The only way to resolve this dilemma is to kick down to assembler, and hand code some x86 that follows the Win32 calling conventions, which are well explained here and here. So here’s the code I wrote to invoke arbitray functions exported from a DLL. I used a couple of great resources to refresh my ASM chops, which have become very rusty after years of neglect: this primer and this x86 instruction set reference. It’s in inline assembler, together with the C++ preamble that sets up parameters to simplify the assembler.

bool cc_cdecl = true;                         // stdcall if false
int parmBytes = ( parmCount - 1) * 4;         // parmCount includes ret val, so subtract 1
int parmPop = ( cc_cdecl ? parmBytes : 0);    // number of bytes to pop off the stack after call
void* rvInt = 0;                              // for receiving int or ptr return value
double rvDbl = 0.0;                           // for a float return value from ST(0)
int paddr = ( int)parms;                      // parms is void** array of parameters. Cast to int
                                              // to prevent implicit ptr deref by asm
// Then asm code to do a cdecl or stdcall dispatch and call xf.
__asm {                      // push parms onto stack in reverse order
        push eax             // save eax
        mov eax, paddr       // point to start of parms
        add eax, parmBytes   // point to last parm
    pp: push [eax]           // stack a parm
        sub eax, 4           // point to next parm
        cmp eax, paddr       // have we hit the start yet?
        jg pp                // if eax > parms goto pp
        call xf              // invoke the function!
        add esp, parmPop     // pop parms if cdecl
        mov rvInt, eax       // int or ptr retvals are in eax
        fst rvDbl            // float ret vals are in st0
        pop eax              // restore eax
 }

The Resolver

February 1, 2008

So I’ve had a little play with Resolver One, now that it’s out of beta and at 1.0. To give a somewhat simplistic view, it’s an Excel clone implemented in IronPython. Where it scores big over Excel is in exposing its internals, specifically the calc model and worksheet in one seamless view. In Excel, we have several different APIs for injecting our own logic into a spreadsheet: Excel’s own internal C API for coding XLLs, VBA and COM. Those APIs allow us to express worksheet functionality in C, VB and any COM language. Resolver gives us one seamless view of its internals in our favourite programming language: Python. Simple example here.

This is attractive for those of us already using Python for front office systems. I’m not sure how much it will appeal to others though…

A new kind of spreadsheet…

September 28, 2007

Excel is unavoidable for anyone working in front office trading systems. Front office support teams often bemoan the hassle of supporting myriad sheets, all a spaghetti mixture of various internal and external plugins and trader coded VB. Whole lines of business get built on spreadsheets, and replacing them with robust apps soaks up many developer years.

But spreadsheets will always be on the trading floor, no matter how hard IT may try to eliminate them. The reason is simple: Excel is the traders’ preferred development platform. Yet at the same time, it’s a general purpose tool, quite ill suited for trading apps. I’ve long thought that a determined start up could make a killing by building a better spreadsheet for the trading floor.

Being a long time Pythonista, and just recently dipping my toes in the water with IronPython for desktop etrading apps, I was excited to discover Resolver Systems. The Resolver is an IronPython .Net based spreadsheet that supports injection of Python scripts. I hope the Resolver guys are thinking about multi threading and serverisation as well as integration with existing .Net based Excel addins. Multi threading and server side capability would allow serverisation of trader coded pricing, and support for existing .Net addins would enable Resolver sheets to plug into market data infrastructures etc.

Good luck to the Resolver guys !  I’m very happy to see a promising startup furthering the cause of Python on the trading floor.

VC8 XLL

March 25, 2007

So if you’re building an XLL in VC8, and you’re using a .def file to specify the exported symbols, make sure you give the linker a /DEF:myaddin.xll switch. Unlike the VC6 linker, VC8 doesn’t automatically pick up the .def. Took me a good few hours to figure that, using depends and filemon to look for missing dependencies etc. All the time depends was telling me that my XLL was exporting no symbols, but I couldn’t see it, so fixated was I on more subtle possible problems.

Sparklines

September 26, 2006

Sparklines is a visualisation technique I’ve been following for a while – so I read this with interest. And then something else on cmiles blog caught my eye: a link to Juice Analytics. They have a nice blog on Excel hacks, which mentions a toolkit that gives you sparklines in Excel. Not the same Juice as founded by Charles Ferguson who wrote High Stakes – probably the best book I’ve read on founding a software start up.

Matlab, Python, R

September 22, 2006

Thanks to Vince for this Matlab/Python/R crib sheet. It’s good to see the three way equivalence, especially since I’m an old hand with Python and an R newbie. Matlab is the tool of choice on our trading floor for traders who’ve hit the buffers with Excel.

Excel 2007 at CSFB

August 1, 2006

Just got an email from MS referring me to the presentations from last Tuesday’s Office 2007 presentation at the Credit Suisse offices. Paul Foster’s session on Excel Services is worth checking out…

Playing with R

August 1, 2006

I’m doing some data analysis at the moment, and I’ve gone through some toolsets in search of a combination that will give me power, expressiveness and charting capability. The raw data comes out of our trading system’s RDB. I started off with SQL queries, but soon found the slowness of complex queries and the arcana of SQL was holding me back. So I exported the data as CSV and set to work with Excel. I made some more progress, but didn’t want to resort to VBA for the ad hoc slice and dice coding. So then I switched to processing my 300Mb CSV with Python. Progress was good for a couple of days, but then my script started to get hairier. The data extraction and cleansing logic was all mixed up with the analysis logic. And I still didn’t have any good charts. Back to the drawing board. I vaguely remembered folk on Victor Niederhoffer’s mailing list discussing R. I checked out the programming recommendations on the great man’s site. So I downloaded R, and after 30 minutes playing, I’m very impressed. Scripting, powerful maths, built in vector and matrix data types, and flexible charting all rolled together. I’ll be asking around on the floor to see if any of our traders are using R…

Excel 2007 again

July 28, 2006

A couple of emails with Paul have reassured me that live pricing sheets driven by real time market data and hosted by Excel services will be possible. The solution is to code an RTD server that uses Excel’s web services API to kick off a recalc that will pull in the new data. Hope the managed code required by Excel Services doesn’t introduce too much latency.

And in other news, congrats to Marc on his new gig!

I went to the Office 2007 event at Credit Suisse on Tuesday night. Jim Burns, CTO financial services Microsoft UK opened the event. A CSFBer called Leslie spoke for a few minutes. He announced he worked for R&D in CSFB, that they identified emerging trends for the next 3 to 5 years, and that virtualisation and MS Office 2007’s distributed work features were the coming things. Then Ian Moulster, Product Manager for the Dev & Platform Group in MS UK spoke briefly. Then finally, we got the meat in the sandwich, an hour long presentation by Paul Foster, Developer Evangelist on Excel Services and InfoPath.

Excel Services is based on Sharepoint, which seems to provide some app server like hosting of the all new Excel Server calculation engine. Excel Server hosted worksheets can be rendered in a browser, and support interaction using AJAX techniques. Excel Server hosted worksheets can be invoked programmatically from a client using a web services API too. Paul alluded to XLLs a couple of times. I asked him about them afterwards. From the Excel 12 blog, it looks like C/C++ XLLs on a client Excel will be similar. However, a server side XLL, running under Excel services, must be managed code, though unmanaged code can be wrapped. When I asked Paul about injection of real time data into a server side worksheet, he indicated that my DDE based technique wouldn’t work for Excel services, as Windows permissioning would disallow it. And now the cumgranosalis blog tells us that RTD doesn’t really work for Excel services.

Why is this a problem ? It’s not unknown for traders to implement ad hoc pricing engines in Excel. I’ve worked on an Excel based system that autoquoted exchange traded option strategies. The recalcs were driven by a real time feed of futures prices – the future being the underlying for the options. We injected the futures price with DDE. It’s looking like it won’t be possible to implement this kind of system with Excel services, which is a great pity.