tag:blogger.com,1999:blog-12948038.post6573278430688861007..comments2023-07-29T09:15:17.416+01:00Comments on allan's blog - Agile & Digital Business: Agile: Where's the evidence?allan kellyhttp://www.blogger.com/profile/06262139490250478379noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-12948038.post-8612927627096521332012-05-04T19:13:42.814+01:002012-05-04T19:13:42.814+01:00Hi Laurent -
You are right that evidence shoul...Hi Laurent - <br /><br /> You are right that evidence should be verifiable and fully auditable, but what about situations like the one I mentioned? I was an employee wanting to know how well my team compared with whatever was known about similar teams. I couldn't open up our code to be examined by others. My managers would not have been interested in inviting a software metrics company in to evaluate our code. Would they have allowed it if it was for free? I am confident the answer would be no, just due to suspicion that doing so could somehow expose proprietary information. Safer to say no than yes.<br /><br /> My fallback was to use a 3rd party static analysis tool to profile our code for complexity, average length of methods, etc. and then include some of that data in the experience report. (It was too voluminous to include all of it.) If you accept that productivity has to do with amount of code per person hour then the next problem is how do you prove what hours were worked? <br /><br /> Again, we get to the problem where I didn't have the authority to make our labor records public. But there are more problems with auditability. For instance, the connection between software statistics and good design is not direct. Without an ability to examine the actual source code. <br /><br /> My earliest validation was from talking informally with other early practitioners, and with Ward Cunningham at the initial Agile conferences. I found several teams that claimed they were seeing only 1 or 2 bugs per month. Conversation showed me they were not some weird corner case, so I took that as useful info. Also they weren't selling me anything;-)<br /><br /> In the end I figured that the odds of a whole team making up a bunch of phony data and claims were longer than the odds that the practices we described were having a positive effect. So I wrote up what I had. <br /><br /> I agree that science demands replication, and that our field has extremely low evidential standards. <br /><br /> I also agree that we need to find a way to have authoritative data supporting claims made. Without that, Agile is too vulnerable to those with other agendas. <br /><br /> But this is a big goal and would need a real effort by a group of us who are serious about finding some practical way for us to surface the evidence and subject it to peer review. Is there a group already working on this? I checked the Agile Alliance site for the list of current programs. There isn't one for this idea. Want to consider starting one?<br /><br /> - Nancy Van Schooenderwoert, @vanschooNancy Van Schooenderwoert, @vanschoohttp://www.leanagilepartners.com/noreply@blogger.comtag:blogger.com,1999:blog-12948038.post-27187668395246762912012-04-10T13:36:36.155+01:002012-04-10T13:36:36.155+01:00Actually PM Hut I have looked around me, in fact I...Actually PM Hut I have looked around me, in fact I've worked on a good few IT efforts which would consider themselves "waterfall" (if they ever had to name it) and actually: I don't think they ever worked.<br /><br />I think you are mistaken. I think if you looked inside the projects you refer to you would find very very very few of them defined what they were about to build, "designed it", implemented it, and tested it without any feedback between those activities.<br /><br />Therefore, I do not believe the Waterfall ever worked. QED.allan kellyhttps://www.blogger.com/profile/06262139490250478379noreply@blogger.comtag:blogger.com,1999:blog-12948038.post-2137063262523458582012-04-10T13:08:04.241+01:002012-04-10T13:08:04.241+01:00Hi Allan,
When answering the question "Where...Hi Allan,<br /><br />When answering the question "Where is the evidence for waterfall?" have you looked around you?<br /><br />I'm saying this because nearly every product around us was built using Waterfall, and every construction project still uses the Waterfall methodology.<br /><br />I suggest you read the comments on this post: http://www.pmhut.com/agile-project-management-a-solution-to-the-changing-project-landscape where one of those who commented claim that Agile projects never get finished.PM Huthttp://www.pmhut.comnoreply@blogger.comtag:blogger.com,1999:blog-12948038.post-32274929146869743202012-04-02T10:26:10.057+01:002012-04-02T10:26:10.057+01:00"First although it sounds like a reasonable q..."First although it sounds like a reasonable question I’ve come to believe that this is a question that is asked by those who don’t believe in Agile, those who want to stall thing. It is rarely a question aimed at a rational decision."<br /><br />Alan, that may have been true 10 years ago when we were basically struggling to get permission to try Agile.<br /><br />It certainly is no longer true today. Agile has entered mainstream discourse and providing evidence is one of the responsibilities that come with claiming the territory of a knowledge discipline.<br /><br />We shouldn't restrict the call for evidence specifically to Agile claims though. That's not how knowledge works - we don't cherry pick the claims we like. Rather, we are all engaged in the work of figuring out what software development is about; what reliable knowledge we can have about it.<br /><br />Many claims that are considered "ground truths" of software engineering, pre-Agile, turn out to actually be pretty shaky. (See my book-in-progress "The Leprechauns of Software Engineering"). IMO software engineering needs a reboot, and I'm hoping that this reboot might come from the Agile community. But this reboot is only possible if it values hard evidence over hype and marketing schemes.<br /><br />For instance, the early quote (1996) about "600% productivity gains" from Scrum: I've actually tried to get at the underlying studies to see what the evidence behind that was. I've contacted Capers Jones (who the quote was attributed to), he doesn't remember writing that and specifically disclaims having seen that kind of productivity gain from Agile. I've contacted Ken Schwaber (on whose site the quote was posted), he referred me to Jeff Sutherland and added that he tries to avoid productivity claims. I've reached out to Jeff Sutherland and he's only been able to point me to studies way posterior to the 1996 quote. This quote should be viewed as extremely dubious; just because someone claims to have measured something doesn't mean the evidence exists.<br /><br />My stance is "if the original research isn't fully auditable, the raw data and methodology out in the open, it doesn't exist". There are any number of ways that you could fool yourself into thinking that you had witnessed a 600% productivity gain. Science demands replication.<br /><br />Unfortunately, our field has had extremely low evidential standards so far.Anonymoushttps://www.blogger.com/profile/07379040637704958977noreply@blogger.comtag:blogger.com,1999:blog-12948038.post-51744633380845135702012-04-02T03:39:24.849+01:002012-04-02T03:39:24.849+01:00Great post, Allan! It's disturbing that people...Great post, Allan! It's disturbing that people seem to mainly use data to support ideas they already hold, not to reason by. <br /><br />Indeed "where is the evidence for Waterfall?" The best answer I've seen is in "Agile and Iterative Development" by Craig Larman. On pp. 102 - 105 he describes how Winston Royce was actually recommending iterative development in his 1970 paper that got commonly mis-read as advocating a single-pass waterfall process! <br /><br />But that's not even the most incredible part of the story. Craig found and spoke with the original author of DOD STD 2167. That's the one that institutionalized waterfall for the US military. I quote: "He expressed regret for the creation of the rigid single-pass waterfall standard…in hindsight, he said he would have made a strong iterative & incremental development recommendation, rather than what was in 2167." <br /><br />I can vouch for some data concerning Agile processes. With my early Agile team in 1999 - 2001, I wanted us to try this set of practices but I didn't want to have only anecdotes after all was said and done; I wanted to have real metrics.<br /><br />At the start of our project, I persuaded the members of my team to cooperate in understanding where our time was going and what results we were getting. We did 3 years of sustained Agile development work for a challenging embedded system. We measured about 3 times more output than comparable teams, and we averaged about 17 bugs per year. There was never more than 2 open defects at any given time. This quality level was far beyond anything I had seen before in software development!<br /><br />I wrote up an experience report on it (given at the Agile conference some years ago) which is available here:<br />http://www.leanagilepartners.com/publications.html<br />The title is "Embedded Agile Project by the Numbers With Newbies"<br /><br />It took a long time but I eventually found credible industry data to compare our numbers with. I found it hard to believe we were that much more productive, but I knew how all our data was created and there was no funny business going on. We simply wanted to understand the reality, not to promote (or bash) any idea or practice.<br /><br />Best of all, others can use the benchmark numbers to see how their teams compare. Some of the most useful data I found was from Capers Jones. He looked at the comparison I did, and agreed that the analysis technique is valid. <br /><br />You're right that it's important for people to go collect their own data and reflect on what it means. Thanks for bringing this up.<br /> <br /> - Nancy Van Schooenderwoert, @vanschooNancy Van Schooenderwoerthttp://www.leanagilepartners.com/noreply@blogger.comtag:blogger.com,1999:blog-12948038.post-47834524002261131712012-03-31T19:38:03.722+01:002012-03-31T19:38:03.722+01:00Jeff Sutherland presented some productivity measur...Jeff Sutherland presented some productivity measures based on functions points in http://jeffsutherland.com/SutherlandDistributedScrumHICSS2007_v6_7_Jun_2006.pdfAnonymousnoreply@blogger.com