Programming Fashion Faux-Pas

We’re releasing a new version of our code today so the last week’s basically been a free-for-all as far as fixing bugs and implementing enhancements. I had to push some additional Swaption data onto TIBCO for Blackrock to consume. So I had the pleasure of modifying an interface I’ve never worked on before. So I come across this:

if (product instanceof Swap) {
    ...
} else if (product instanceof TotalReturnSwap) {
    ...
} else if (product instanceof CDSABSIndex) {
    ...
}

Dude, are you kidding me? You ever heard of an interface? I think that right there tells me whether or not the person who wrote this code is a seasoned developer with good refactoring skills. I should remember to use that as an interview question. Frankly I wouldn’t hire a guy who codes like that…. It shows a complete lack of programming fashion sense.

Work’s been keeping me quite busy lately, actually. We are finally going live with CDS on ABX next week and I’m pretty pleased since I did pretty much 95% of the development for that. I got to get very intimate with the various credit events that can occur and how they should be handled. Actually Calypso’s not quite up to par in how it models CDS on ABX, at least as far as accounting is concerned. For accounting to get what it wants, a separate transfer needs to be created for each individual credit event. So if MarkIT says on a given month that ABX.HE.BBB-.06-1 has got an interest shortfall of 59.2, an interest reimbursement of 12.5, and a fixed correction of 15.1, Calypso needs to generate 3 individual transfers. As of 8.0.3, Calypso doesn’t even handle corrections at all. (Maybe they fixed that in 9.0?) Of course, the individual transfers can and should be netted together, but each one should generate its own transfer type and, further downstream, its own posting.

Anyway, it was interesting making this enhancement and seeing it trickle all the way down to Peoplesoft.

Think Outside the Box

I came upon an interesting problem late last week.

I’ve recently implemented an OTR Bond interface to Calypso so that, when new TIBCO messages alerts us of a new “On-the-run” Bond issue, we can insert the appropriate data into Calypso. Of course, if it’s a new Bond, there’s also a good chance that we don’t yet have the Bond definition in Calypso and, for that, we use the Bloomberg Connect API to retrieve the Bond and link it to the Calypso BondBenchmark… Sorry. That was just an overview but I can already see that most of y’all have your eyes all glazed over. Didn’t mean to bore you to death.

Anyway, I’m rather familiar with Calypso’s Bloomberg Connect API and how it works. Puh-lease… give me a hard one!!! After all, I pretty much wrote the whole thing single-handedly a couple of years ago. Let’s just say that a request file is generated, then FTP’ed over to Bloomberg. Then, it’s your responsibility to poll the remote directory for the equivalent response file. Once it is available, you download it and parse it locally with the data you requested. Pretty straightforward, right?

Well, when I implemented the Bloomberg interface, I thought of the “requested” scenario and never bothered (foolish me!) to think outside the box. Let’s say you request a Bond with CUSIP 123456, then I’d simply generate a request file called cy_123456.req (cy being short for Calypso. Y’see? To top it all off Bloomberg states that there’s a maximum number of characters in the file name. Some puny amount like 12 or 15, I think.) So anyway, cy_123456.req would get processed and FTP’ed to the remote directory. Then, I’d poll every once in a while for a file called cy_123456.out. When the file would be present, I’d FTP it back to my local directory and remove the file remotely, thus cleaning up after myself.

This ain’t rocket science, right? That should work just fine, no?

If you’ve got one instance of Bloomberg Connect running, it runs fine and dandy, yes… but here at Countrywide, we got a bunch of environments all running in parallel. We’ve got a couple of development boxes, UAT, and of course, the production environment. The configuration for these boxes are identical and the OTR Bond adapter task kicks off from 4 AM until 5:30 AM. So now you have these 5 or 6 processes all running on different boxes each listening to TIBCO. Let’s say at 4:05 TIBCO broadcasts: The 3 month OTR Bond has CUSIP 2223123. Well what’s been happening is this: these processes would check Calypso and determine it that Gosh-darn-it… we ain’t got that Bond definition. Let’s go fetch it!

5 processes on 5 different machines each generating a request file, cy_2223123.req. Some time passes and the Bloomberg Engine running on each box processes the file and uploads it to the remote (COMMON!) Bloomberg directory. Some time later, these 5 processes would each look for some file, cy_2223123.out, download it back locally and proceed to delete the remote file.

Hmmm…. can I get semaphores implemented on top of FTP? On second thought, don’t answer that…

I’m surprised that the process failed so infrequently, but occasionally, you’d have some poor process who’d never get its .out file. No wonder, some selfish sibling would have wiped it out before he got a chance to retrieve it. The funny thing is that, had I even pondered the question back at Calypso, it would have easily been apparent. I just never bothered to think: “Hey, wait… what if they run this process 5 times instead of just once, which seems logical.”

The quick fix? We’ve decided, until Calypso issues an official fix, to stagger the execution time for OTR Processing. Typically TIBCO will broadcast OTR information sometime between 3 and 4 AM and, of course, that information remains cached for new subscribers. So as long as we don’t connect before 4 AM and we have each box start processing OTR Bond messages on TIBCO at 10 minute interval, each instance of Bloomberg Engine should, hopefully, get to do its thing by itself. It’s not the most elegant solution, but hopefully it’ll do the trick!