Sunday, November 27, 2016

Technical Liabilities and not Technical Debt

My last - frustrated - post finished with this line:

“banks are collecting technical debt the way they used to collect sub-prime”

I’ve always disliked the tech debt metaphor, in part because the way it is generally used is different to Ward Cunningham original defined technical debt and in part because those using it often have a simplistic understanding of debt - for example, credit card debt (which is generally to be avoided) is different from mortgage debt (which many of readers are only too glad to take on.)

A couple of years ago Chris Matts and Steve Freeman suggested a better analogy: an unhedged call option. While technically correct this metaphor required one to understand financial markets and options, most people outside the financial arena (and a good few inside it!) aren’t familiar with such concepts and so the idea floundered because the metaphor was more difficult to understand than the thing it was describing.

Still, Chris and Steve instincts were right, especially in financial arena’s the tech debt metaphor may be doing more harm than good. For a banker collecting debt is good… let me explain.

For you and me, as individuals, a debt is something I have to repay, it is a call on my future cashflow. For individuals - and many non-financial companies - a debt is a liability. As such we would rather not have it.

But for a bank a debt is an asset.

The bank holds the debt, when I take out a loan I promise to pay a bank sums of money in the future. Therefore my liability is their asset. It is two sides of the same coin.

Conversely, for a banker liabilities are things they own other people, e.g. my savings. When I place £1000 in my savings account it is an asset for me but a liability to the bank because the bank need to pay me £1000 at some date in the future. (Importantly, I decide that date and am unlikely to forwarn them.)

Bankers want more debt because debt is an asset. Debt is good.

So every time as software engineer says to a banker: “Doing it this way will create more technical debt” the banker hears “Doing it this way created more assets.”

One of the problems that occurred in the 2000’s was that banks found a new way to take on even more debt. The debt was packaged in technical ways “collateralized debt obligation” and “tranches” and such which allowed the banks to assume more debt. Things went well for a while and bankers believed they had found new way to make money. The problem was these debt packages were hideously complicated, so complicated in fact that many of those trading in these instruments didn’t understand them.

Nor did the banks own risk officers.

Nor did the regulators.

Does this sound familiar? Isn’t this technical debt?

What happened next made things worse: new instruments were divised by some of the people who didn’t understand how the original instruments were structured. In Fools Gold Gillian Tett tells of how the original JP Morgan team who devised these debt products warned as other banks introduced products which they could see didn’t make sense.

Doesn’t that sound familiar?

Doesn’t that sound like the Chief Engineer telling the Captain “She cannae take it, Captain” and the Captain doing it all the same? (Did Star Trek condition those in command to ignore the advice of their engineers?)

Banks aren’t unique here but their business model makes the problems more acute. Even outside of banking the technical debt metaphor has encouraged “debt thinking”, i.e. the idea that one can borrow from the future to get something delivered sooner and pay back later. This might be viable if we treated such debt as a mortgage which allows an asset to be purchased now in return for a long term payments plan.

However when programmers use technical debt to borrow from the future it is more like a payday loan which very quickly balloons and demands increasing amounts of our capacity (cashflow) to service the loan.

Banks are an extreme example of debt funded businesses. Most companies balance the amount of debt they take, if they need more money they may borrow it or they may ask shareholders. In general reasonable to see companies as getting half the money they need from shareholders and half from borrowing.

But banks don’t.

Banks may get less than 10% of their funds from shareholders, they rest they borrow. Anat Admati and Martin Hellwig in Bankers new Clothes explain this in detail but basically a non-financial company couldn’t do this because there is a high risk the company would go bust. But a bank can do it because in the event of failure they believe (and RBS proved) that the government will save the bank.

In other words: banks pile on debt because there is someone else who will save them if something goes wrong. (Economists call this “moral hazard.”)

It is a common business practice to borrow (increase debt) to improve returns so bankers aren’t alone in seeing financial debt as good but they are extreme.

So the question is: Do bankers believe they can take on technical debt because someone else will save them?

I can’t answer that question but clearly engineers view debt differently to bankers. Its more complex than a simple failure to understand.

Bank IT runs on a project model: as I say in #NoProjects this model encourages cutting quality because of goal displacement and the erroneous idea that it will be “done.” Even if bank managers don’t believe it will ever be “done” they can see their involvement ending, perhaps they are contractors, or perhaps they will move onto another “project.”

There are plenty of companies out there, be they outsourcers or tool vendors, who are only too happy to say their service or tool will solve the problem:

“Tech debt too high? No problem for Far Eastern Outsourcer No.1, we can fix it!”

“Do you suffer with long testing cycles? Then get Cyber Tester 3.0, it will find your bugs and shrink the cycle quicker than you can say contingent convertible bond

As Willy Sutton is might have said: “I sell IT to banks because thats where the money is.”

So is there a way out of this?

Well, there is one more complication that might actually offer a way out.

Another dimension to banking is time: the loans that banks make (their assets) extend over the long term (25 years for a mortgage) and the bank have little power to force repayment. They are said to be illiquid - it is difficult to cancel an existing loan so you can’t turn a loan you have extended into hard cash very easily.

But, the deposits banks accept (their liabilities) are very liquid. I can get my money out of a cash machine at any time and leave the bank with less money - ever wondered why banks pay higher interest rates on fixed savings?

Some say it is this problem that banks exist to solve: turning short term liquid deposits into long term illiquid loans; matching assets and liabilities on different time scales. (Mervyn King has as interesting discussion about this in The End of Alchemy.)

When we build a software system we are building an asset.

The system itself - retail banking systems, trading platforms, risk management systems - are long term investments and become major assets. Some last decades but they are illiquid, changing systems is hard. RBS has recently aborted a demerger because of the difficult of building a new retail system (FT: RBS chief warns it may fail to sell Williams & Glyn this year).

Conversely, defects in these systems (bugs) and poor architecture (what is often called technical debt) are liabilities. More importantly, like banks financial liabilities, these can strike at any time. I can visit the cash machine today and an unknown bug can crash the system today. Similarly, a poorly designed sub-system - say there is a singleton in the system - might not crash a system but can slow down operations, enhancements and so on.

Sure these problems might not appear today, but they might, likewise I might not visit an ATM and withdraw cash today, but I might.

Therefore, let me suggest that we drop the language or Technical Debt. (Or perhaps let Technical Debt return back to Ward’s original meaning.)

Lets instead talk about Technical Liabilities:

Technical liabilities, in software, are sections of code, even entire systems, which because of their (poor) design are difficult to change, significantly hinder enhancement and are fertile ground for defeats. Such code often lacks unit tests and coherent design.

Such liabilities are highly liquid and problems may transpire at any time - impeding performance of the system itself or enhancements by programmers.

In building an asset some liabilities will be incurred. But,

  • a) it is possible to minimise these liabilities while building and
  • b) these liabilities can be reduced with investment.

Reducing the liabilities will make the asset more valuable itself and enhance the asset’s ability to change in future.

Replacing the words “technical debt” with “technical liabilities” is a small change to our language, one we can all understand easily, removes a poorly understood metaphor that is open to misinterpretation.

Wednesday, November 02, 2016

Banking systems stink - the pain of an botched HSBC release

I’ve had a very frustrating morning, and it all comes down the failure of bank IT systems - particularly a software release failure at HSBC. Gee, their systems are flaky.

I was trying to buy tickets at for my trip to Lean Kanban France in a few weeks. First I tried to use my American Express card (I hate it, I only use it for work expenses) and I got the now traditional “Service not available -Unhandled Java exception …gavity.exceptions…” at the final payment stage - you know the pocky little box you have to put three letters into? - I’ve had this error before elsewhere and I think its because I block pop-ups on Firefox. Annoying but nobody at AMEX care so that is that.

So I repeated the transaction using Chrome and my Mastercard - issued by HSBC but branded John Lewis.

And again at the final security page - Arcot’s 3D Verified by Mastercard bit - the page errored. Not a horrible unhandled Java exception but an error just the same.

So I called Eurostar - because something similar happened a few weeks ago and they took my booking by phone. Only this time they insisted that it was the banks fault - like two banks fail? Didn’t anyone ever tell them “Select isn’t broken”?

This time they insisted the problem was with the banks, to make a booking over the phone would cost me £10 - something they waived last month. I even went as far as to speak to a supervisor who continued to insist he wouldn’t waive the fee - which I know from last month he has the ability to do.

Instead he told me it was probably my Mac that was the problem.

Seriously, he told me to use a phone instead. 10 years ago saying “don’t use a Mac” might have been acceptable but now?

Bad customer care on both points.

I gave up, I fell back on Safari and another card altogether to make the booking.

After several tries and a long wait I finally got to speak to HSBC Mastercard and this is what I discovered…

  • HSBC Mastercard don’t currently use Arcot security but intended to start on 9 November
  • But the release has been pulled because of technical difficulties, they don’t know when it will occur
  • But the message had gone out to retails that they were to start (remember, today is 1 November so they change wasn’t scheduled for 7 more days)

This also explains a couple of other recent online purchase problems I’ve had with this card.

This implies that either:

  • When a seller system (e.g. Eurostar) calls the HSBC system (“is_using_arcot_security()”) the HSBC system is replying “True” when it should say “False”


  • There is a piece of system configuration that HSBC publishes to sellers (“<is_using_arcot_security>True</is_using_arcot_security>”) which has been pushed out prior to the update and that a) this config has been released too early (its not 9 November yet!) or b) else has not been rolled back.

Either way HSBC have bungled a software release. Of course, it might not be HSBC, it could be Arcot, or rather their current owner CA - who were fingered in the RBS outage a couple of years ago.

Which also might explain why I spent over 30 minutes on hold to the call centre (plenty of failure demand).

And we still haven’t discussed by HSBC pulled their release in the first place, clearly something went wrong to cancel the 9 November release - my money is on low quality.

When will these companies learn: high quality in software is faster and cheaper.

Continuous delivery is about risk reduction.

(Well, they are probably mainframe systems so maybe I should be a little forgiving on the second point.)

Now someone at HSBC will probably get into trouble for sharing so much information with me. Well its a good thing they did because I wasn’t going to hang up until they did. Thank you, you did the right thing - unlike Eurostar.

I write this both to relieve my frustration and in the hope that it might make HSBC (and AMEX) do something about this.

(I should also point out: none of these companies have ever hired me. If they had I’d probably keep quite because I avoid blogging to specifically about clients.)

But you know the really worrying thing about all this:

Our modern banking system is based on the assumption that the Government will always bail out a failing bank - the fear of the payments system failing is so great governments will do anything, thats why we rescued RBS.

Now it seems banks are failing to maintain the payments system anyway. The banks are collecting technical debt the way they used to collect sub-prime. Central bankers throwing money around won’t work.

Who will save us this time?