Scrum + Technical practices = XP ?

My last blog post (Scrum has 3 Advantages Over XP) attracted a couple of interesting comments - certainly getting comments from Uncle Bob Martin and Tom Gilb shows people are listening. And it’s Tom’s comment I want to address.

But before I do, lest this come across as Scrum bashing I should say that I think Scrum has done a lot of good in the software industry. Yes I had my tongue-in-my-cheek when wrote “Scrum has 3 advantages over XP” but if Scrum hadn’t made “Agile” acceptable to management it is unlikely that XP ever would have take Agile this far.

I suppose I’m a little like Uncle Bob, its a bit of a surprise to me that Scrum Certification has come to mean so much to the industry when really it means so little. I’m also worried that brands poor people “certified”, good people “uncertified” and stifles innovation.

Second the ideas of the Scrum originators are good. And lets not forget that while Schwaber and Sutherland have become celebrities in the community there were five authors of the original “Scrum: A Pattern Language for Hyperproductive Software Development”, Beedle also co-authors one of the early Scrum books, Devos still works as a Scrum Trainer, while Sharon seems to Scrum’s Fifth Beatle - where are they now?)

Getting back to the point. Tom Gilb commented: “Sutherland did say in a speech that Scrum works best with XP”. Chris Oldwood also says he’s heard Jeff say this.

In fact, I think I’ve heard Jeff Sutherland say something similar, heck, not similar, the same and reported it in this blog. And Jeff’s not alone, I’ve heard the same thing from other Scrum advocates, trainers, and cheer-leaders.

But, this view is not universal. There are Scrum trainers who don’t. Specifically there are those who train Scrum and specifically say XP practices like TDD and on-site customer should not be used. And as I sarcastically commented previously, part of the attraction of Scrum to management types is that it doesn’t delve into the messy world of coding.

Lets just note: there are multiple interpretation of what is, and is not Scrum. Nothing wrong with that, it just complicates matters.

Back to XP though....

There are, broadly speaking, two parts to Extreme Programming (XP). The technical practices (Test Driven Development, Continuous Integration, Refactoring, Pair Programming, Simple Design, etc.) and the process management side (Iteration, Boards, Planning Meetings, etc.)

It is these technical practices that Jeff, and others, are saying Scrum software development teams should use. Fine, I agree.

Look at the other side of XP, the process side. This is mostly the same as Scrum. Do a little translation, things like Iteration to Sprint and it is the same.

(OK there are a few small differences. I just checked my copy of Extreme Programming (old testament) and to my surprise stand-up meetings aren’t there. I think many XP teams do stand-up but it isn’t originally part. The book does include “40 Hour Work week” which isn’t present in The Scrum Pattern Language, or Agile Project Management with SCRUM but I’ve heard Scrum advocates talk about sustainable pace. I’m still not sure how you marry this with commitment but I’ve written about that before. However these points are knit picking.)

Essentially Project Management XP style is Scrum.

This shouldn’t be a surprise. In the mid-90’s when Scrum and XP were being formed and codified many of these people knew one another. If nothing else they saw each other’s papers at PLoP conferences. The Scrum Pattern language was presented at PLoP 1998, Episodes (the basis for XP) was at PLoP 1995, and both draw on the work of Jim Coplien.

I have every reason to believe that XP and Scrum didn’t appear in insolation. There was cross-fertilisation. I have also been told there was competition.

If you accept that Scrum and XP project management are close enough to be the same thing, and you accept Jeff Sutherland’s recommendation (namely do XP technical practices with Scrum) then what do you have?

• XP = Technical Practices + Scrum Style Project Management
• Jeff Sutherland says effective development teams = Scrum + Technical Practices from XP

Think about that. Even talk about it, but don’t say it too loudly. For all the reasons I outlined before, XP isn’t acceptable in the corporation. Scrum is. Therefore, we must all keep this little secret to ourselves.

Scrum has 3 advantages over XP (but XP is better)

For corporations Scrum offers three advantages over Extreme Programming:

1) It offers certification - corporates can hire people who are “certified” and have their own people certified.
2) Scrum traces its roots back to the Harvard Business Review - so it must be serious
3) Scrum does not contain the words “extreme” or “programming”. No need to dirty our hands with messy code, keep that stuff as far away as possible (the further the cheaper, right?)

On the other hand, XP has two advantages over Scrum:

1) It actively seeks to address the quality problems most software development teams suffer from and which cripple productivity
2) It excites the people who actually do the work - dispensing with “resistance to change” in one fell swoop

At Agile Cambridge last year I saw a really good presentation from people at GE Energy about their Agile adoption. Frankly I’m sceptical about the ability of any large organisation to adopt Agile but these guys had some real success to show.

Two things stuck in my mind. When summing up the presenter said:

“If I was doing the same thing again I would start with the XP technical practices not the Scrum process” and he went on “We have adopted Scrum, we are now advancing to XP.”

XP and Scrum date from about the same time (mid-90’s). Perhaps because XP become popular first and Scrum later succeeded it as the Agile-poster-child many people seem to think Scrum is superior, or at least a superset of XP. Actually, the reverse is true.

XP is a superset of Scrum, and, in my humble opinion, superior.

Software Facts - well, numbers at least

About a year ago I needed some numbers about software development - industry norms really: effectiveness, productivity, bug counts etc. etc. Its actually pretty hard to get these numbers and after hunting around I found myself with a copy of Capers Jones Applied Software Measurement.

Jones, for those who don’t know, has made a career out of analysing software and software teams numbers. Applied Software Measurement is his latest work on the subject - although its nearly three years out of date.

Unfortunately, for me, Jones didn’t have the numbers I wanted - I forget what I wanted now. But he does have lots of interesting facts and numbers. You can dispute some of these numbers, you can argue with him but on the whole he knows more about this subject than you or me so he’s probably right. That said, there are a few points where I disagree with him and I might blog about these in future.

For now, I’d like to share some of the number in Applied Software Measurement. Overwhelmingly Jones researched American companies and American teams. I expect most of these numbers will be broadly the same in Europe and elsewhere. All figures are for 2008 unless specified.

Productivity

• Broadly speaking, in the USA, software productivity rates were the same in 2008 as they were in 1988
• The best US companies in terms of productivity and quality are about three times better than average (US)
• The worst US companies for productivity and quality are about 50% worse than average
• Only about 15% of companies are improving productivity and quality
• Similarly, about 15% of companies are getting worse in these terms
• About 70% of companies are neither improving or getting worse
• Where productivity is improving it is usually teams developing web applications who use Agile methods. Some benefits also accrue from use of Team Software Process and Personal Software Process
• Claims by tool vendors of 10-to-1 or 20-to-1 displacement of human activity are generally not justified. It is counter productivity to invest in tools before resolving organisational and methodology issues. i.e. save the tools until you’ve improved things (John Seddon should be happy with this finding.)
• Office environments can have as much of an impact on productivity as tools and methods
• Looking after employees well increases productivity and reduces turn-over
• Companies providing 10 day or more of training per employee per year have higher productivity rates than similar companies who do not
• Accounting errors (and use of unpaid overtime) can hide the true cost of software production by 100%, i.e. the work costs twice as much as the accountants say
• Formal design reviews and code inspections are effective but can fall into disuse because (new) managers do not understand this. (Tom Gilb is right!)
• “Software does not age gracefully, and tends to become increasingly unstable and difficult to modify safely. The Unites States and indeed much of the world are facing some huge geriatric problems with their inventories of ageing and decaying applications” - maintenance, renovation and enhancement are now the dominant activities in the industry and it is likely to stay that way
• On average military software is 25 times larger than end-user software, and the specifications and other documentation 200 times larger.
• Productivity and quality seem to be better in object oriented languages

Documentation & Bugs

• Producing paper documents for software development is more expensive than producing software itself
• Up to 400 words may be written in specification for every line of code in large systems.
• In a large software project requirements change and increase at the rate of 2% per month
• Repairing defects in software is more expensive than coding and producing documentation
• The finding and fixing software defects (bugs) is historically the most expensive activity and will account for 50% of total costs over the product life time
• Approximately 7% of changes and fixes contain new defects
• Most forms of testing finding fewer than 30% of all bugs

Projects

• Over 50% of all software projects are one-man projects
• Productivity and quality are directly coupled: projects with high quality have high productivity and vice versa
• IBM was the first company to discover the link between quality and productivity. They made the information public in 1975 but it is still not widely know.
• User satisfaction and defect counts are also directly correlated

I’ve mined these numbers out of Applied Software Measurement, I might mine some more and blog again. But really, it isn’t a book of numbers. Its a discussion of how to measure software and software teams.

Finally for now, I’m only at chapter 3, Jones says that currently available data on software production is less than 1% of what is needed. He also says that most databases of such data are privately held and not accessible by most of use. Well, I guess that explains why I couldn’t find the numbers I wanted when I wanted them.

Announcing EuroPLoP 2009 Proceedings in print

A long long time ago, well summer 2009 I was Programme Chair of the EuroPLoP conference. One of my responsibilities was to produce the conference proceedings. For the first time we published these proceedings online using the CEUR repository.

Now another first. I am pleased to report we - well me actually - have made the proceedings available in print using the Lulu system.

So, roll up, roll up, get your EuroPLoP 2009 Patterns Conference Proceedings for just £13.54 in print, or free as a PDF.

This is something of an experiment. In the past EuroPLoP had printed proceedings but these were only available if you actually attended the conference or knew someone who did. This limited their circulation.

Secondly, the proceedings were paid for out of the conference budget which added something like €100 to the price of attending.

Whether future EuroPLoP conferences continue to publish on Lulu, or CEUR, is a matter for debate. If you have a view let me know.