Could the Credit Crunch be a software fault?

The Financial Times is reporting that the credit rating agency Moody’s has uncovered a bug in the software they use to value credit derivatives. As a result some instruments rated 4 notches higher than they should have been. And Moody’s shares are down 15% as a result.

Think about this. If these instruments had the correct rating people would have put a lower value on them to reflect the risk. And banks would have had a different value at risk so maybe they would have seen problems coming sooner.

How come Moody’s didn’t notice this sooner? Well it seems they did but they didn’t want to face up to it - management not wanting to hear what engineers say? - Sounds like the Challenger disaster again.

What happened to code reviews? What about Unit testing?

Let me guess, ‘this is the real world, we don’t have time for that’. The financial industry is build on software but banks haven’t really internalised that yet. The Moody’s bug is not an isolated instance.

Story 1, I was working at an investment bank in the late 1990’s. I was working on a risk system with the quants. They had branched the code base from the main developers months before - when I say branched, I mean it was put on another server so there was no way back.

The bank was silly enough to loose its development team - five of them left within a few weeks of one another. I was asked to take over the main code base and support the main overnight risk assessment system. There was a list of bugs they wanted fixing, most of which I’d fixed in the other code base, but before I could do so I needed to build the live system.

The two code bases on two different servers were different, not just diverged but contradictory. The code left on the developers machines was different again. And being C++ there were a lot of #ifdef’s and I couldn’t tell which ones were on when the live system was built.

I had no way of building the system to produce the same numbers as the live one.

I asked for help but nobody really wanted this news. I could build a system, several versions of it in fact, but none of them produced the same figures as the live system. And nobody had any time to help me work out what the figures should be, or even tell me how I could work them out.

The bank’s risk analysis software was effectively a random number generator.

Story 2, I heard this at SPA this year from “Bill” - a pseudonym. I think it was a derivatives or risk system but I’m not sure. Bill had been examining some figures one evening and they didn’t look right. The more he looked the more wrong they looked.

Bill found a developer and together they looked at the code. And in the code there was a sign error. Numbers which should be positive were negative and vice versa. They made a fix, and reported up. It went all the way up the chain and fast. The fix was signed off and in production very quickly.

I didn’t find out if the bank ever reviewed the numbers, or how long the error was live but, it was quite possible the bank was insolvent on several occasions without knowing it.

Moral? Increasingly our financial markets are built on systems which are not only difficult too understand but which we known contain errors. So what happens now?

Most likely Moody’s will survive - even if their shares fall. The moment the word “code” is used management will tune out, nobody wants this news so lets move on.

But if it is just possible the SEC, FSA and other regulators will get involved. This could be serious and they might demand something is done about it. Not just at Moody’s but elsewhere - clean up financial systems!

Now this is a more interesting scenario because two things might happen.

Firstly this could make Sarbanes Oxley regulation look like a walk in the park. What if the SEC introduce their own coding standards? What if all software needs to be formally approved?

This route would be a field day for the CMMI and ISO guys, the high priests of formal methodology and for document management systems. They would have so much work... and banks software would grind to a halt.

It would also be good news for vendors of shrink wrapped software. If developing your own costs so much more, and gets you tied in regulation then it is no longer cost effective so buy, don’t build.

There is another option, far less likely I’ll admit but...

What if you had an audit trail? You could build the system for any date at the push of a button.

What if code reviews did happen? Or what if code was pair programmed?

What if all source code was unit tested? And the tests kept? And the tests re-run regularly?

What if writing code became so difficult, expensive and risky we had to reduce our use of code? Well then we’d have to really code the right thing - improve our requirements.

Do you see were I’m going? Improving our code base, even regulating it, doesn’t have to be SarBox all over again. Many of the Agile methods are surprisingly well suited to this.

We will see. In the meantime, next time someone tells you you are wasting time writing a unit test remind them that Moody’s lost 15% of its value because of a bug and may have endangered a fair few banks.

Do we really learn from failure?

Its an often heard expression: “We learn from our failures”. Particularly when you’ve just failed at something: “Well put it down to experience.”

I’ve always had my doubts: if we can learn from failure can’t we also learn from success? But how often do you hear people say: “Great success! Now what did you learn?”

All too often we don’t stop, examine our failures, take time to reflect and actually do the learning from them. We’ve failed, failure is painful, we want to put that behind us, forget about it. Maybe we could make time but do we really want to? Who wants to dwell on what has gone wrong? Naturally, after a failure we are defensive, and when we are defensive our learning process aren’t at their most effective.

In the IT world failure isn’t an absolute. Its not like a football match, one side goes home knowing they scored more goals than the other and therefore “won.”

If an IT project delivers late is it automatically a failure? What if it succeeds in the other goals?
What if it is late, and costs more, but delivers more value to the business?

I often tell story I found when I was writing my MBA dissertation. For this project I interviewed a variety of software developers about past projects, one guys, lets call him Paul (because that’s his name), he said:         ‘The project was a great success’

He went on to describe how the technology was good, the product did what the customer wanted and everyone was pleased with the outcome. Then I said: ‘You say the project was a success, but before that you said it took 12 months to complete, and how it was originally estimated to take 3 months. Some people would say a project that was nine months late on a three month schedule was a failure’

To which he replied: ‘I never thought of it like that’

Failure is ambiguous. So how do we know when and what to learn?

In Wednesday’s FT there is a piece by David Story in which he looks at some statistics. He discusses the failure rate of entrepreneurs. If we learn from our failures we should expect entrepreneur who have failed in business to be more successful if they launch another business. But it turns out the chances of a business succeeding are the same whether the founder is a first time entrepreneur or an entrepreneur with a failure behind them.

You are as likely to success with your first business as your are the second or third. The odds don’t improve.

The only thing that does change is that you try again. If you never launch a business venture you can’t have a successful one. If you failure once you can’t have a success unless you try again. He likens it to buying tickets in the lottery.

If we don’t learn from our failures, we don’t learn from our success, and often we don’t even know what was successful and what was a failure, then how are we to learn?

We have to buy more lottery tickets. We have to institutionalise our learning, we have to create routines to learn and set our learning triggers.

Product Management conference

On Tuesday I attended a mini-conference about Product Management. In fact, this conference billed itself as “The UK’s first conference on Product Management” which might actually be true.

I’ve written before that I believe Product Management is a much misunderstood role in the UK, and that I think it is an appreciation of this role that makes Silicon Valley companies so much more successful than their UK peers. Well, this conference might be a sign that things are changing. The fact that it existed at all, that it had sponsors, and that it was full is a good sign. However, I still came away with a feeling that only a few companies “get it.”

Practically the conference was well organised and free. It was sponsored by Telelogic (part of IBM) and Tarigo - a company I’ve not heard of before but I’m glad to say offer Product Management training in the UK. As with any vendor sponsored conference it was in some ways a marketing exercise by these two companies, but that said, the marketing wasn’t too heavy and they didn’t ram their products down your throat.

However, the conference was pretty tightly controlled, no questions until the end and no involvement of the attendees. Quite different to my usual type of conference, but then ACCU, SPA and PLoP’s just aren’t ordinary conferences, I’m spoilt.

So what did I learn?

Well I had my enthusiasm for Product Management warmed up again - it would be good to work as a Product Manager again!

Tarigo presented some estimates they had done of the number of Product Managers in the UK. Much to my surprise they believe there are about 90,000 to 110,000 Product Managers in the UK, which equates to one Product Manager for every nine developers.

On the face of it this is positive. I had no idea there were so many! And the ratio of 1:9 isn’t too far adrift of my own recommendations on what you need (1:3 for innovative rapidly changing products, 1:7 for less innovative more stable products) but, Tarigo has included Business Analysts in their figures.

I need to write a full entry on Business Analysts and Product Managers. In some ways the role is similar, in others it is different. I think most BAs could benefit from learning about Product Management and I think that is why Tarigo included them. But, on the other hand the roles are different. In the Q&A session I asked the panel about this and they agreed the roles are different.

So I think Tarigo’s 1:9 figure is optimistic.

And to prove the role is misunderstood I spoke to people at the conference who had nothing to do with technology. They are a different type of Product Manager again. Looking at the attendee list and their companies I think more fell into this category than I realised.

I also got some ideas for my growing business pattern language and confirmation of some of the things I’ve been writing in my latest set of patterns.

I learned a little about Telelogic’s product - I didn’t try to hard - and it confirmed another pattern I’m seeing: specialist database products. I first noticed this when I came across ChimSoft - Chimney Sweep Software - honest, as far as I can tell they are for real.

What I see is this: databases are wonderful things, they can do anything! But to use them effectively you need to program them. In any sector - chimney sweeping, product management - there are people who need a database and don’t have the skills to programme FoxPro or Excel let alone Oracle or DB2. Therefore, there is a market in specialised databases in specific sectors. And exploiting it is nine-tenths good product management because there really isn’t any new technology here.

Extend this argument upwards, to more complex domains, and you arrive at CRM and ERP systems. Both of these are in many ways databases with fancy utilities. But again, this is another blog entry.

Back to the conference.

Finally, perhaps the most surprising thing at the conference was the reaction to one session about Agile Software Development. Unfortunately the speaker didn’t link it up with Product Management very well so it was basically a “This is Scrum” talk but the audience got very very interested. Most of the Q&A session was about Agile in fact.

So why?

Well I think there were several reasons.

Firstly most of these people had not been exposed to the ideas behind Agile before. This just goes to show how things have yet to take off.

Second: the Agile story is very compelling if you know a little about software development. Agile promises products on time, with quality and everyone is happy.

The problem with the Agile story is: unless you know a little about software development already then it doesn’t sound any different to anything else. You abstract the message for those who don’t know software development (i.e. senior managers) and you get: improved productivity, reduced time to market, increased business value, etc. etc. At which point are we talking about Agile or Ration Rose? Or .NET? Or BoundsChecker? Or outsourcing to EDS? Or India?

I have more insights from the conference - so it was worth going to - but they are destined for my current crop of patterns.

London Software Partners - a new venture

Ever since I launched Software Strategy I’ve nominally offered training in software development, specifically Agile Software Development and even more specifically the management and process of Agile Software Development. However, I’ve never actually pushed this side of the business.

Today we are launching London Software Partners to offer such training. I say we because this is a joint venture with Giovanni Asproni.

London Software Partners arises from two trends we have noticed. First is the difficulty many of our colleagues have in applying Agile development when working in banks and other financial institutions.

Second is our belief that Agile development is at a critical point in its development. Agile is going mainstream. Many Agile practices, for example TDD, are now see as nothing unusual. Unfortunately a lot people are doing a pale imitation of Agile which creates more problems than it solves.

So London Software Partners intends to address these problems by specifically focusing on the application of Agile Software Development techniques in banking environments. Too many Agile courses - and books - throw out general advice. We intend to give specific advice for specific problems.

We will offer some general public training courses but our main focus will be on tailored training for specific customers. This we plan to solve specific problems. “Solutions through Learning“ is out motto.

Giovanni and I have decided to tackle this together because, well, two heads are better than one. Although we are being very focused it is still a pretty big problem to address. Our skills complement one another: he is certified Scrum master, I’m a certified PRINCE2 project manager; he has spent the last few years with his head in the code, I’ve spent them managing teams and so on. We believe that to offer a complete solution you need to tackle both sides and together we do that.

Anyway, check out the website and let us know what you think.

Make strategy like you make software?

There is an interesting piece in the latest issue of the MIT Sloan Review entitled: Should you build strategy like you build software?

I can imagine some managers initial reaction: What? IT is such a total disaster why would we want to make strategy the same way?

After all, we are regularly told that 70% of IT projects fail, and a few months ago the same journal ran a piece damning software development and specifically ERP systems: The Trouble with Enterprise Software

So why would anyone want to copy what IT does?

Well, believe it or not there is something interesting what the author, Keith R. McFarland is arguing is: many of the practices and techniques used in Agile software development can be applied to strategy formation and execution. McFarland focus on techniques such as small iterations, collective ownership, overlapping phases, direction changes (i.e. refactoring), organising around people not tools and abolishing big up front design.

To some degree I think he’s a little behind the curve. The myth about long range / strategic planning in companies was exploded a long time ago. Sure some companies still do it but that doesn’t mean it works. Henry Minztberg’s Rise and Fall of Strategic Planning explains why it just doesn’t work and why its a myth.

It is not only software development were managers and companies have suffered from the Illusion of Control it occurs in strategy formation and planning. Strategy formation is an emergent process, in the same way that software design is emergent.

Big strategic planning has been out of fashion for a while but its far from clear what should replace it. McFarland seems to be suggesting the answer is Agile Strategy.

There is an irony in McFarland’s work – and that of Donald Sull at LBS - is that while businesses thinkers, and some businesses, are looking to Agile Software Development for a paradigm of devising strategy too many businesses are still resisting Agile development or finding it impossible to implement.

Agile development is still largely a bottom-up change exercise. Not enough senior managers are embracing and backing Agile, they cling to the illusion of control. Too many people still say ‘Agile is unproven’. The fact that McFarland and Sull are prepared to embrace these ideas for strategy formation shows that Agile ideas are valid.

Regular readers will know that I have been arguing that there are close parallels between business strategy and software development for some years now. This argument is embedded throughout Changing Software Development. It is most clearly seen in the early chapters were I discuss the knowledge and learning aspects of development work. (The argument is also embedded in my MBA Dissertation “Software Development as Organizational Learning” if you want a less polished - more academic - version.)

And I have written in Overload about it – An Alternative view of design (and planning).

Most directly I’ve blogged about it on and off, specifically last August
Agile software and strategy and Thoughts on strategy and software development.

But this is more than just a parallel: for companies which use a lot of technology software and strategy are increasingly converging. Ultimately your software is your strategy – so much so that I sometimes image software code as liquid strategy.

There are two forces at work here. Firstly, as much IT - including software - becomes commoditised (Nicholas Carrs argument) what remains is strategic. Your Word Processor is a commodity, but your inventory management system is strategic. if you inventory management system is a commodity then your customer management system is strategic, and so on. Because IT is so varied, and software so capable - limited by your imagination - it is used to embed our knowledge.

Which is our second force: what we know. We embed our knowledge in our code so our organisations can operate: whether it is the Galileo booking system, Google’s Adwords or Unilever’s ERP system the capabilities and limitations of our IT systems are also the capabilities and limitations of our organizations.

As Cynthia Rettig argued in her Sloan Review piece, the limitations imposed by an ERP system impose costs on organization and limits on what they can do. That in turn limits on strategy options.

At the same time software systems open up opportunities and create strategy options for companies. We see this most directly with companies like Microsoft and Amazon but it is also true of companies we might not expect. Companies like FedEx depend on software in order to be able to execute strategies like delivering packages on time.

Banks are on the cutting edge of this convergence, unfortunately that doesn’t mean they do it well, but they are converging. Go to the financial centre of London or New York and you’ll probably find more software developers than investment bankers. Trading derivatives, managing risk and countless other banking activities would be impossible today without the sofwtare.

So where does this leave us?

Companies which embrace Agile in software development can learn a new way of working which will help them elsewhere – specifically in strategy formation

Companies which embrace Agile will be better positioned to as software and strategy converge

No company has to embrace Agile, you can do it the old fashioned – see IT as a cost outsource it, run it like a train timetable. But those companies who embrace Agile will win. The more you embrace it the more you win.

Presentation to BCS and IIBA in Cambridge online

I spoke at a combined meeting of the British Computer Society (BCS) Cambridge branch, International Institute of Business Analysts and Cambridge Assessment on Wednesday night. Actually I was only one half of the act, Paul Turner was the other half.

It was an enjoyable night, Paul is a long term Business Analyst who used his presentation to describe the role of business analysis in an Agile development cycle, and I used my session to explain why Agile works better with some Business Analysts or Product Managers involved.

Download my presentation here: Agile Practices go better with Business Analysis, or Why Business Analysts and Product Managers are more important than ever for Agile Software Development