E-book: The Secrets of Market-Driven Leaders

Regular readers of this blog will know I hold Product Managers in high esteem. When developing software the presence of Product Managers and the use of product management techniques is one of the keys to success. Unfortunately there is very little literature on the role of a Product Manager.

At the moment, the best way to learn about Product Management is simply to go on one of the courses offered by the likes of Pragmatic Marketing, Silicon Valley Product Group or 280 Group. I went on the Pragmatic Marketing course a few years and am really glad I did. Highly-recommended. If you want to do a course in the UK your options are more limited, Co-herence is the only one I know.

If you don’t have the time (and money) to go on a course then your thrown onto reading. So far my standard recommendations are: Crossing the Chasm and The Inmates are Running the Asylum. Neither of these is about product management but they do cover some of the necessary ground. (And if you are a Product Manager they are essential reading.)

Well, the people at Pragmatic seem to have a book on the way. In the meantime they have made available an e-book called The Secrets of Market-Driven Leaders. I have not had time to read it in full yet but what I have read I like. For anyone trying to work out how to manage a product (or what Product Managers are) it might be the best thing yet.

High Church, Second System effect and re-writes

After ACCU last last Thursday I was in the pub with Bronek Kozicki - and many others. The subject of system re-writes came up. This is a perennial subject, for example have a look at Alan Griffith’s Overload 79 editorial or Tim Penhey’s CVu editorial from the same month.

Recently I’ve been consulting with a company which has a truely awful code base. The developers want to re-write it, naturally. Unfortunately the the code base isn’t the only problem, the company has other problems and matters aren’t helped by Kissinger-esque linkage with other issues. I’m not going to say anything else about them except to say Conway’s Law was clearer than ever.

And its not just software people who stuggle with the question: Is it better to re-wright, or not to re-write? This piece from last week’s FT shows English lawyers and tax authories have the re-write problem.

Bronek sent me a link to Portrait of a Noob by Steve Yegge - not a blogger I read but I’ve been sent several of his posts by people in the past so I know of him. In many ways I find this reminiscent of my own High Church C++ pieces from a few years back. Both pieces point out that quality code is subjective, what you think of as good code in one context is not the same as what you consider in another context. Steve is right to point out that your persepctive on code changes. In particular, as you get older and acquire more experience not only does you coding style change but the way you approach problems changes.

As someone pointed out on Thursday night, you also need to remember Fred Brook’s and the Second System Effect:

‘This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.’

I’ve long wanted to conduct a little experiment. Take a developer, take a system he developed in the first few years of his life. Wait long enough for him to forget the details, say 10 years. During that time let him learn, gather experience and continue developing systems. Now ask him to make changes to the system from his early years - with all the identify removed of course. My theory is, he will proclaim it unmaintainable and in need of a rewrite.

That is a little unfair actually because one of the things you learn as you get more experience - OK, get older, lets not beat around the bush - is that all systems are maintainable. The other thing you learn is that rewrites usually end in failure.

Our education system has it wrong. At school and college we learn to new write new code. But most developers spend most of their time maintaining old code. Our education trains us to believe a lie, that we write new systems. For the first few years of a graduate engineer’s career he is disappointed, he searches for ‘green field’ development and faced with an old system he throws his arms up - ‘Re-write! We must re-write!’

Not only are young developers sadly disappointed by the state of the world they are a risk. Sometimes people believe them and let the re-write the system. (Believe me, I was once that developer. I persuaded my employer to let me re-write a system. It was a mistake. )

Colleges need to recognise this and start setting assignments were people must maintain code written by others. Systems should be handed from one year to the next for changes.

As a developer gets more experience they learn how to work with existing systems, when to re-write parts and when to work around. The hallmark of an experienced developer is that they do not jump to say ‘Re-write!’ - they can at least present options.

In fact, the hallmark of the best developers is that do work with legacy systems. Anyone can write a new system, we get trained for it in college, its what the books describe, there are few limitations. You can create your own mess, and you usually know your way around your own mess.

But the best developers can work with someone else’s mess and still find gold.

Spoke at ACCU London last night

I spoke at ACCU London last night. I little unusual because I’m usually the one organising the talks not giving them. But I’m not just a talk organiser, I have things to say myself - as anyone who reads this blog knows well! And, in case I didn’t mention it, I have a book to plug! :)

The subject of my talk was ‘Agile development - where to begin’. Essentially it was intended to be the same talk I gave to ACCU South Coast and ACCU Cambridge before Christmas. I’ve now uploaded the 2007 version of this presentation (the previous link) but, and this is a big but, the presentation I gave last night was very different... let me explain. (Stick with me, this finished with me learning something and being grateful to the CEO of an investment bank.)

Normally ACCU London meet at the offices of 7 City, a training company which specialises in financial training. We are really grateful to 7 City and we are gradually finding ways in which the ACCU can repay their hospitality.

As it happened 7 City were fully booked last night so couldn’t find a room for us. So we looked around for an alternative venue. Giovanni Asproni arranged for us to have a room at Barclays Capital, in Docklands. This was a good venue and reminds me that ACCU London should visit Docklands more often. Anyway...

Then Giovanni e-mailed to say ‘By the way, you won’t have a projector’ because Bob Diamond (CEO of Barclays Capital) was using all the projectors the company had. Don’t ask me how he was using them but he was. So, I would have to do the presentation without a projector.

Now I’ve given presentations before without a projector or PowerPoint, and in truth I don’t particularly like PowerPoint. But it is a crutch I lean on. And I think people expect it to be there. And it gives me something to post on the web afterwards.

I gave a talk at the BBC before Christmas without PowerPoint and projector but that was different. I deliberately aimed for a more conversational style. But to do a straight through presentation? 90 minutes?

Well I revisited my presentation and drew some mind maps to remind myself of the key points and to ensure I kept a consistent argument and went for it. And, I think, I am told, I pulled it off!

In the end I was more relaxed doing the presentation, I could take detours into stories, I wasn’t faffing around with a keyboard and remembering what the slides said. And in the pub afterwards several people told me they not only liked it but preferred it that way. I am now resolved to do this more often - and you read it here first.

I’ll still use PowerPoint and projectors sometimes - I have already written my presentations for SPA and ACCU 2008 - but I’m going to stop using it. And for this I have to thank:
• BarCap for providing a room
• Bob Diamond for using all the projectors
• And probably the alignment of the planets

As I said, its not often I thank the CEO of an Investment Bank, indeed, this might be the first such time. Bob pushed me out of my comfort zone and I learned something.

Unfortunately for you dear reader, I have nothing for you to download. If you want to know what I said you have two options: buy the book, or invite me to give a talk to your organization/company/family.

Book review: IT Success!

I’ve just finished reading IT Success! (yes, there is an exclamation mark there) by Michael Gentle and I’m glad I did.

Before I write on, and before you read on, I should declare a connection. I share a publishing house (John Wiley & Sons) and and editor with this book so you might consider my opinions biased. However I had no idea this book was being written and produced at about the same time as mine. The first I knew about this books’ existence was a mention in the FT. That said I think it supports a lot of the arguments I make in Changing Software Development.

IT Success isn’t about Agile development but it does assume that you can deliver software in an iterative, Agile, fashion. With that assumption in place the author spends most of the book considering the business side of software development and project management. The book is really concerned with in-house and bespoke software projects so don’t bother with reading it if you work on the ISV side of things.

Although the book is not about Agile development I think it says a lot that can help organizations looking to improve their delivery by using Agile techniques. Most writing about Agile concerns itself with the software development process, some goes as far as project management but little goes further. This is where Gentle starts really, as such he fills in a lot of the missing bits about (strategically) managing iterative based development.

Gentle starts from the same assumption as I usually do: corporate IT, specifically software development, is fundamentally broken. He ascribes this to the use of the construction metaphor to IT project management. That is, the idea that one party can specify what they want and contract with another party (in house or external) to supply it. In this I completely agree with him for two reasons: first you can’t know what you want in advance because learning takes place during development what you want changes and usually increases.

Secondly: you can’t write down everything you want. Suppose for a moment you could, then you would have to write down a lot of detail, however this much information would overwhelm the person reading it. Either way you can’t communicate what yo want.

But back to Gentle...

He also identifies a mismatch in supply and demand: because most IT is funded centrally there is always more demand than there is supply. Think about it, if you are not paying for something, or you get it cheap, then you will want more. Cost acts as a demand regulator.

Gentle solution is two fold. Firstly to properly apportion costs to reduce demand and secondly to change the funding model for corporate IT projects. Instead of the current budget allocation process he suggests using a model more like ISVs use with the venture capital backers.

To do this each project produces a business case with defined benefits. This is funded for a short period, at the end of it the cost-benefit analysis is repeated - or rather it is continual - and if the benefit is still present more funding is allocated. In effect we move funding onto an iterative model as well as development.

Of course there is more to it than this but I’ll leave you to read the book. The book is short and written in a casual style so is quite readable. Personally I would have preferred a slightly more formal style, I don’t like the frequent citing of what could be casual observations or asides at conferences. I would have liked more research and cross-referencing but that may be personal preference and it would have detracted from readability.

I would also have likes more evidence that pieces of the solution suggested work. I’m sure Gentle has seen them work but too often they come across as ‘good ideas’ and it is not clear to what degree this has been tried and tested.

There are a couple of points I disagree with Gentle on too. To take one, his calls for the extensive use of time keeping by development staff as part of his billing policy. I find his description naive. For starters there is some evidence that time tracking by all staff - not just developers - is hopelessly inaccurate, on study I saw (sorry, reference lost) said it was 150% inaccurate.

More to the point if staff must keep track of their time it assumes they are working on more than one project at a time, this in itself is hugely inefficient. It is better to assign someone to a project for a week, a month, a year so they get to know it and they are not task switching. After that you can block book all their time to that one project and be done with it.

Anyway, I still recommend this book, especially to those people who are charged with commissioning and managing corporate IT projects. It is another contribution to changing the current, broken, model.

For all my IT contractor friends in the City: Rates are up

Over the last few months there has been a reoccurring topic at any gathering of IT workers in London: What does the turmoil in the financial markets, the so called ‘credit crunch’ mean for employment?

Specifically: are we looking at another down town like in 2001/2002?

Well, the answer is, according to this piece in last weeks FT we have an answer: more money.

The average hourly rate has increased from £45 to £50 per hour. Of course most of the banks moved away from hourly rates a few years ago, most people earn ‘professional daily rate’ so multiply that by 8 (the official number of work hours in a day) but expect to work 9.

The City of London sucks up a lot of IT talent, and many of the best people choose to work as IT contractors. This is logical, the City pays well, jobs come and go but there are always projects, they just move around. Since IT is always at the bottom of the bonus pool you might as well take your bonus one hour at a time.

Frankly I think most of the IT work in the financial markets is pretty boring, the City pays well to offset the boredom factor. Most of the problems I have seen or been told about in the City are down to:

• No shared goals between departments, call it political infighting if you prefer couples with:
• Individuals who will not share their knowledge: traders, analysis, quants and yes, especially IT people
• People who don’t understand IT and a continued belief that IT is like any other resource to be managed the same way. Thus leading to:
• Adherence to the traditional project management model

Add in an extreme short term view which occasionally swings to extreme long term and you have a recipe for disaster.

I don’t think many financial institutions actually care about getting their IT right. After all they make money from financial deals not IT. The IT can be pretty screwed up and you can still do trades and deals. They won’t go bust from bad IT so you can afford to carry on doing it wrong.

However, banks and friends do have large amounts of money, so when they do care about their IT they can afford tools and advice from snake oil sales men. Whether they are selling tools, services and silver bullets, none will fix things, at the end of the day you really have to care to want to fix things.

Myth busting

With my last blog entry, The Location Myth, I realised I am slowly building up a list of software development myths I don’t believe in:

• The Location Myth
• The Documentation Myth
• The Requirements Myth
• The Egoless Programming Myth
• The Design & Planning Myth
• The Plug Compatible Programmer Myth

I like this idea... maybe I can find some more, any suggestions?

The Location Myth - why location it matters

The internet, we were told, would make location irrelevant. But it hasn’t. Some examples.

• My brother lives in the UAE so we talk by phone. We would use Skype but the local telephone company blocks Skype and various other services.

• I have several kitchen containers from a New Zealand company. I bought them in the UK and would happily buy more but they are no longer stocked in any shop I know - and believe me I’ve tried. I could order them online from the US but the American shops won’t ship outside of the USA and Canada.

• When I’m searching the web, particularly if I’m looking to buy something, and I find the kind of site I’m looking for I want to know where are they based? London? Cardiff? Chicago? Mumbai? I have nothing against India companies, or American companies (provided they will ship abroad) but when I’m faced with a site I’ve never heard of I want some kind of reassurance. Their location is part of that. OK, there are fraudsters in London as there are in Chicago, I’m unlikely to chase them myself, and they could be lying about where they are, but it all adds to my understanding.

In short: the internet makes many things possible over long distanced but location still makes a difference.

The same is true in software development teams.

You can have people located in different offices, different parts of the world, and you can use telephones, wiki’s, Skype, video-conference kit, etc. etc. to work but it isn’t the same as having everyone in the same place.

When you locate people elsewhere you incur extra costs. Obviously the cost of telephone calls or bandwidth but these are a small part. As you use more technology (conference calls, video conferencing, shared whiteboards, etc. etc.) you incur more and more direct costs. Even wiki’s take time to installl, and time to use.

You also incur costs because conversation often need to be arranged. So I might IM someone and say ‘Can you talk?’ and 20 minutes later they get back and say ‘In 5’. The need synchronise before talking slows down the process and there is a cost of asking.

Then there is the cost of missed conversations. We make small talk, trivia, at the water cooler, or walking to lunch, or because someone is there and we want to moan. When someone is remote these conversations are lost. We incur cost because we don’t get the benefits of these costs.

Then there is the cost of setting up meetings to compensate. A few years ago Lise Hvatum and myself ran a focus group at EuroPLoP entitled ‘What do we think of Conways Law now’. One of the recommendation we came up with was: when you remove an informal communication channel you need to replace it with a formal one.

So when we have lots of remote workers you need more formal, arranged in advance, meetings. This ads to the costs again.

And when you do talk the level of information transfer is less. Even with video conferencing your primary communication mode is voice. Communication is centred on explicit words. Face to face much communication is implicit, its in the tone of the voice, the facial expressions, the body language and how other react. You loose all this when you are remote.

Lets face it, software developers don’t have a great reputation for communicating well anyway so why put obstacles in the way?

Every time we put an obstacle in the way we reduce information transfer. This goes for cubical walls and offices too, open plan offices are my preference. Every barrier, cubical partition, different floors, different buildings, different continents, reduces information flow. The bigger the barrier the bigger the loss.

And for a development team - focus on that word: team. Who ever heard of a football team which played on different pitches?

Team work is about just that, team work. You can’t do it when you are alone. Every barrier makes team work more difficult.

It is an illusion to think that you can develop software when your team is spread around a single building let alone the world.

This is especially true for people who are in leadership positions in an organization. You can’t have a CEO, a team leader or an architect who is removed from those he leads.

Sure there are exceptions. A few companies are completely distributed, Sleepycat software - before Oracle bought them - springs to mind, as do Open Source projects. On the whole these work because they are born on the internet.

The other reason remote working teams work is because they aren’t so much a team but a bunch of individuals, each working on their part of the system and seldom interacting with each other. I think of these as Swordfish teams - a bunch of developers flying in formation. As a result (Conway’s Law again) you are likely to end up with a very diverse system architecture.

Don’t get me wrong, I’m a big fan of working form home once in a while. The odd day spent working at home can be really productive - precisely because I have few conversations and can focus on one or two things. Working from home is great for reflecting and preparing oneself.

And it is acceptable for large teams to have the odd person working remotely. But the smaller the team is greater the need for them to be in one place.

Maybe once upon a time developers could work alone, remotely. They could put on headphones and switch off to the rest of the world. But today developing software is a team sport and needs to be played like one.

I say that as one who has never liked team sports such as football, rugby, cricket, etc. I’ve always preferred solo sports like skiing, running and swimming. I have to adopt and change too.

Developers are not the only fruit

My younger self would be horrified to hear me say this but, when you develop software you need people who are not software engineers.

Don’t get me wrong, coders/engineers/programmers are the most important people because these are the people who actually produce the thing. As such they are central to any development effort, you can’t write software with a collection of managers, analysts and testers any more than you can build a ship without shipwrights.

Coding is where the metal is bent, shaped and welded. When coding there is no room for ambiguity or fudge, the code doesn’t lie and it doesn’t take to ambiguity. Thus there is a lot of details in coding which aren’t obvious before you start. So your developers need to be able to call on Product Managers for more detail and, if you have any architects, you need them at the code face coding with your developers.

See Jim Coplien and Neil Harrison’s patterns Work Flows Inward and Architect Also Implements for more discussion of this topic.

One of the worst projects I ever worked on was completely top heavy with managers (programme, project, customer, delivery, team - you name it we had a manager for it), testers, analysis, admin staff - yes you needed a lot of admin staff when the project was over 120 people. But only about 12 actual coders. Crazy. The overhead was there because... well, the ISO-9000 procedures needed a lot of managers, and it gave the consultancy implementing the project a lot of bodies to bill for, and it gave the Government (the people who were ultimately paying for it all) a lot of reassurance that it would be done one time, to budget and spec. Except, it was late, the spec trimmed left, right and centre, and if there ever was a budget we blew it.

So you see I’m no fan of carry extra weight on a project.

But, you need people who don’t code.

I’ve written before about the role of the Product Manager and why product management is important, and why you need them. So lets say we accept that Product Manager are needed.

In an idea world we wouldn’t need software testers either. And I expect one day to work somewhere where there are no testers. I don’t believe this is an impossible dream. But with the current state of play we need software testers in most organizations.

One role I don’t think we need is Software Architect. On the whole I think most architects are mislabelled. They should be Product Managers, Business Analysts, Senior Software Designers/Engineers or even Project Managers. Sometimes this is because the organization misuses the title, sometimes its because they don’t understand it (after all, what is software architecture?) but most of the time its because of title inflation. You couldn’t give someone a pay rise so you gave them a nice title.

Although I don’t read Joel Spolsky very often I do agree with him about Astronaut Architects. The moment your architects stops coding its time to fire them. Once they stop coding they are a burden not an asset.

Project Managers are another group I don’t think we need. I accept teams need some leadership but Project Managers are completely the wrong people to. Project Managers are trained to analyse, dissect, use whips (badly) and emit hot air. Their are not leaders by nature or training.

The Agile world is re-inventing project management as coaching but unfortunately the Agile world is pretty messed up about coaching. Coaches are useful when you they get it right and they are genuinely improving the team rather than acting as Project Managers with a different title.

But, and this is where my younger self will have a fit: you do need someone to oversee and manage the whole development process. The big problem is very very few people can do this job right.

By way of explanation to my younger self: done badly this role is worse than useless, done right it can really improve things. Unfortunately most people do it badly.

Its done badly for several reasons.

Firstly, most Software Managers have never been trained in the role - or management at all. They ‘make it up as they go along.’

Second, some of these managers don’t understand software development, they come from outside the domain. Such people try and make software development fit their existing model of management but it never will. Therefore they are constantly trying to make a square fit in a round hole.

Third, many of the managers who come from inside the domain, and know how to develop software can’t give up coding software. To be a good software manager you have to stop being a coder. You have to learn to look at the world differently, you have to realise you can’t intervene to fix problems, you can only arrange things so other people can fix them.

When I was younger I once worked for a bank. The code I was working on was terrible and I had no support from the organization. I really really wanted my manager, and his manager, to come and code with me. I thought their big mistake was to stop coding. I was wrong.

Their biggest mistake was not to stop coding but to stop communicating with those who did code. The big mistake was to put too much space between themselves and the building process. Had they involved themselves more they would have better understood the problems and would have been able to arrange things so I could fix them.

If you are managing a software team you should be talking to them every day. You should be improving their working conditions every day. You should be allowing work to flow to them.

When you move from coding to management you have to give up part of yourself, you have to trust the people you manage. It is their code now, not yours.

So, in conclusion: it is wrong to think you only need developers to create software. It is also wrong to think you can code and do other things - you need the separation. But it is also wrong to have too big a separation.

Make any of those mistakes and you just moved one more duck out of line.

The lining up your ducks theory of software development

Software development is easy. Don’t listen to what people say. I was 12 when I started programming and I picked it up no problem. Businesses are full of people who programme without even knowing it: Excel is programming by another name, and how many people write macros?

Yes, programming is really easy. You can buy ‘Programming for Dummies in ...’ - well just about any language you want. People can, and do, create small world changing programs without ever really being taught how to program.

That’s why software development and new businesses go hand in hand. Its one of the reasons why so much innovation today appears as software.

But....

Software development is also one of the hardest things. In fact, developing good software is so complex it might be the most complex thing man has ever attempted.

Moon landing? - it used software, compared to the complexity of the systems in the international space station I’m sure Apollo looks simple.

Nuclear power? - compared to the maths used by the finance industry today it looks simple. And that is all in software.

The thing is, writing a small piece of software can be (but not always) very easy. Problems set in when you want to grow the software, want more people to use it, want to sell it, want to package it, want to re-use the code or ideas, take bugs out of the system, add features, make it run 24x7.

It is very very easy to write simple software and have it do something useful. It is an order of magnitude harder to get it to commercial quality and keep doing it. And, to make it harder, there is no single accepted method for doing this and getting it right every time.

It sometimes amazes me that we get any of this right. More amazing is that software which is fundamentally flawed works, or at least seems to work. Stuff which by any engineering criteria is broken is used by companies every day, they even base their business on it. (There is another blog entry in this.)

Developing good software, delivering on time, producing with sufficient quality, etc. etc. - all the things you expect from ‘professional systems’ is a matter of getting your Ducks in a Row. What I mean is...

Creating good software, meeting expectations, delivering on time and the rest is not just a matter of getting one thing right. It is a matter of getting many things “right” - or at least workable. You can get some of these wrong and still delivery something - maybe a little late, or lacking features but you will do something.

Each time you do something badly, each time one of your ducks is out of position, you won’t break the whole thing. This isn’t a chain, instead of breaking you will reduce your productivity, or move away from the target, or reduce quality. With each duck that moves out of position it gets worse but it still works, somehow.

Put it another way. If creating ‘the best software ever’ means scoring 100%. Each time you move a duck out of position you remove 10%. So:
• don’t co-locate your developers, loose 10%
• start work without talking about design, loose another 10%
• work to a fixed specification or work to no specification loose 10% as well.

Down to 70%.

• Fire developers half way through the project, loose 10% for lost productivity, loose 10% for moral.
• Projects looking late, add more developers, loose 10% for violating Brooks Law.

40% doesn’t look good does it?

But there aren’t many case where you need to score 100%. 90% will do, even 80%, 70% often, some customers will be happy with 40% - or less.

The people who get upset most by this?

Not customers, they get annoyed but quite often there is nothing they can do about it.

It is the developers themselves. They see each loss, they know where all those 10% are being lost. And these guys are problem solvers, they want to fix these problems.

Now go back to my falling off a log theory of business.

I said: you do something right. Not 100%, maybe you just need 10%. But that one thing is about doing something useful, something other people want. Its about benefit and effectiveness.

Viability of your software product/company =
        benefit of your software
                x
        effectiveness of your organization

If your software doesn’t bring many benefits then you need an effective organization to stay in business. You have to be better than the rest.

If your software brings lots of benefits then it doesn’t matter how ineffective your organization - shoot the ducks! - you will stay in business.

If your software has few benefits and you can’t deliver, then Game Over.

Which leaves... lots of benefits and a highly effective organization. Then you have a world beater.

Unfortunately few companies fall into the last category which leave most people working in one of the first three.

'Changing Software Development' is published!

About 5 years after first thinking I could write this book, about four year after I first worked out what the book would look like, and 14 months after signing the contract my book, Changing Software Development is now published.

You can now buy it at Amazon, or from the publishers John Wiley.