Here is an observation which I’ve been been playing back to clients for a year or so now:
- There are two types of company in the world, those who talk about automated testing and those who do automated testing.
- Those who talk about automated testing buy tools like Quick Test Pro, WinRunner and Quality Centre and Rational Test Workbench, Rational Quality Manager, i.e. expensive products from the likes of HP and IBM.
- Those who do automated testing use tools like Selenium, Fit & Fitnesse, Cucumber (inc. Gerkin, RSpec, etc.) and other Open Source products they don’t need to buy
- There are a few, very few, companies which don’t talk about automated testing but on the whole everyone thinks it is a good idea and those who don’t do it would like to
- There are a some companies who talk about automated testing and have no tools; this is a better position to be in than talking and spending money
What I would really like to know is: has anyone else seen this? Or, can anyone give a counter example?
I think there is even a logic here. The IBM and HP products are expensive, so expensive you have to ask the price (actually IBM does give the price of Test Workbench at $5,500 for a single user license). Consequently they need to be sold, it also means that they need to be bought at a high level in the corporation. The people who are sold these products are very disconnected from the day-to-day work of developers and testers. This means the products might not be suitable and even if they are then developers and testers need to adopt them: they aren’t part of the change decision.
In addition: I just don’t think these tools are any good. To be fair, I’m no expert on testing tools but I’ve seen Quality Centre in action. Quality Centre (QC) isn’t an automated testing tool, its a very expensive test tracker. Testers write their test scripts in natural language, e.g. English, into the tool, the tool runs on the side and as they execute the tests they click Success or Fail. At the end QC gives a report.
To my mind QC is a fancy version of Microsoft Word. English language test scripts, if they are any good, are so detailed that they take almost as long to create as automatic scripts but they are expensive to execute because they are manual.
QTP and similar products are often linked to the OS or browser. They result in fragile tests which break easily when a box moves a few pixels or the browser changes.
HP also follow a razor blades model, I’m told: once you’ve bought QTP you need to buy a plugin for every browser to OS you want to use it with. Every time a new version of the browser comes out you need to buy a new plugin - or so I am told, tell me if I’m wrong.
The net result is: these tools are high maintenance and easily fall into disuse if they are ever used.
On the contrary: Open Source tools are adopted by people who do the work because they see the benefit they bring and they don’t need budget or signatures to get the tools. People get them because they want to use them not because a Salesman comes around and convinces the boss that you need a tool.
And because many of the Open Source tools require developers to create glue code to interface the tool to the application they lead to a more collaborative style of working.
(Perhaps some companies who just talk about automated testing have downloaded some of the Open Source tools but never use them tool. Since they are free nobody notices.)
So there is my theory. Anyone else seen this? Anyone else explain it? Anyone got a counter example?
Hi Allan,
ReplyDelete+1 for your post on QTP/QC.
For those who are doing automated testing, like I do, you may look at RobotFramework
Laurent
Seems to be correct. My company has QC installed, but I don't see any one using it. Also users of QTP complained and ran away from it.
ReplyDeleteAt 1e we use load runner for a very specific type of performance testing: exploratory. It's not automated in the sense you mean. We have a specialist who knows the software in and out and uses it to find the bottlenecks in an exploratory way. He creates scripts so that he can repeatedly run the performance test.
ReplyDeleteOnce we identify and remove the bottlenecks we build it into our usual soak tests. These *are* automated, through team city / powershell and custom tools.
I think the word automated has a different meaning in the context of tools like this. Automated means they can easily be repeated. But doesn't mean that they will live for the life of the software and be maintained. They are employed (though not necessarily designed) in a way that makes the cost of keeping them running very high. This is because the people who create these tests are not the people who create the software. Therefore you don't get the benefit of opportunities to change the design of the product to lower the cost of maintaining these kinds of tests. Instead, the tests work from the outside of the system which tends to be very brittle.
I have a friends who train and consult with these tools and I don't hear them talking about these tests becoming part of continuous integration or deployment. I'll ask next time I speak to them since I have never asked this question explicitly.
It is now a bit under two days since I published this blog entry. I will confess to being a little disappointed with the response. The announcement of the blog entry has been retweeted about a few times so that is good, and a few people have made comments on twitter which are in general agreement, so good some other people see what I see.
ReplyDeleteBut, and this is the reason for my disappointment, so far only one person, Jez Higgins, has said “No, we use expensive testing tools and we do automate.” (And that is with Silk Test, not one of the tools I mentioned but still a tool in the same category, i.e. expensive.)
Really, I was hoping for more people to tell me I’m wrong.
Please, give me some examples to prove me wrong!
It is not so easy to reply here: after the first refusal, I've got that I need to write in my WordPress blog name (not the user name), even I logged in my blog in the same instance of the browser). Having said reprovingly that my OpenId (I have none) couldn't be verified, after several attemps and after I wrote in the captcha, your engine took the comment for moderation.
DeleteI'd suggest you post reply here as an anonymous or a blooger, not related to your id. :)
I'm no tester, but as a developer I have a feeling you might be right. It's just a feeling though, based on very limited experience.
ReplyDeleteOk Allan, I can give you my experience. We have a relatively large (8MLOC) code base that is relatively void of microtests, but there are several thousand "automated-through-the-ui" tests that the QA team uses as a firewall against the barrage of fast and loose changes that have historically been delivered to them. It is slow and the feedback loop is painful but for sure the QA team couldn't do all those tests manually.
ReplyDeleteNow the challenge is to improve the internal quality of the software and get the developers to take responsibility for making automated tests for their features and changes.
One of my clients uses Rational Robot to automate their testing of Windows software. Yes, the tests can be fragile, so if anything chanegs then they can need to update quite a few tests, but on the flipside it does flag when things have changed in ways the dev team didn't necessarily think of.
ReplyDeleteThe first project they used it on, they found as many bugs in Robot as in the software-under-test, but it has proven its worth over time. It works well alongside unit tests, as final "system tests" through the UI, testing what the end user actually sees.
Sorry, I can only subscribe 100% of what you said.
ReplyDeleteMy experience:
I have been involved in the merge/acquisition of few software companies and can tell you that we always fell in one of the following cases:
- no automation at all
- Quality Centre used only for bug trucking
- Quick Test Pro, WinRunner were used for very few tests, not systematically
When we firstly introduced automation, we started building internal tools for testing our software.
Since we started adopting scrum we have using Robot Framework. It's really nice. We have integrated with Robot all our previous artifacts and tools and creating rich libraries in a very straightforward way. It's just great for creating Acceptance Tests. But also good for deeper ones.
Some of my guys are starting using Selenium too.
I'm not a user of QTP, however, I investigated into its possibilities (I create test frameworks so I should know the terminology the leaders use and I watch sometimes the logic of such products by playing with trial versions).
ReplyDeleteI would say that times when paid test applications simple clicked by coordinates are gone. For Windows, anybody can build a framework or tool working upon MS UI Automation (C#/.NET UIAutomationClient, C++/COM MSAA, any .NET or COM-supporting language).
Sometimes, clicks by position are needed, in rare cases. Clicks are needed are as to click on a control inside another control or click on a part of a control that is not available to MS UI Automation (old Win32 controls like Wise setup's tree item). In such situations, there is no problem with click as they aren't obliged to be tied to screen coordinates. You can try UIAutomation.codeplex.com as the simplest example of control recognition and sending a click to a handle with or without coordinated position.
Okay, contemporary test tools rarely use coordinated clicks and broadcasting of send keys (keys can be sent to a handle or a control can be send foreground before sending keys, in those rare situations when programmers programmed a text box so programmatically that the whole app falls if MS UI Automation touches the text box, activating the ModifyChanged event :)).
If you have more money than those five to ten thousand bucks for HP, you may also buy a plug-in. For example, Infragistics's WinForms UltraGrid does not support MS UI Automation (as Sheridan controls never supported). It's a hell to automate such grid using UIA (it works very slowly as you need to enumerate all rows, get text from then and click on each that meets your condition). For mere $3k you can buy HP OR IBM RFT plug-in ($6k for two) that works natively with a great number of Infragistics controls. If your project is using one or two controls, you can try automating it by hands. If your project is a suite with a lot of such controls, what are those $3k? A month's work of a tester, whereas automating of 3rd parties with home-grown can easily take months and results are as I wrote above (I personally spent two or three days on a grid and several days on several types of clicks, including invoking a context menu, and grid automation is just minimal: export data from rows, select/deselect rows, etc.).
QTP is a tool that 'software testing as a service' companies love. It is good on 'stream testing' when your products change one by one for significantly short time. It's not as agile as Selenium, however, fixes are published regularly and new browsers, controls, and services are supported after fixes are installed.
This post is probably where I got the most useful information for my research. Thanks for posting, maybe we can see more on this.
ReplyDeleteAre you aware of any other websites on this
testing-tools
If you did any amount of research on any of the QTP development/support sites, you would see that there are, in fact, a large number of companies using automated testing tools - QC/ALM, QTP, BTP, etc. I don't know where you came up with the mistaken impression you have, but maybe you're only working with small, fortune 1000-2000 companies instead of the bigger, fortune 100 companies where automated testing is an important part of QA as a whole. These companies have complex applications that typically touch multiple applications and relying on some developers' unorganized hit-and-miss or strictly manual unit testing would be a disaster just waiting to happen. Personally, I would question the management of any company that would drop $10k/seat on *any* piece of software and allow it to stagnate on a shelf. If the tool is there, learn how to implement it and use it properly instead of whining "it's hard!" I've worked for three companies so far that have these products and if new, still-in-college interns can become functionally profient on QC/ALM in less than an hour, one would think seasoned professionals should be able to pick it up in a similar time frame...unless the real issue is that they are unwilling to do so, being resistant to change. Maybe they'd prefer to go back to using their old 12" green-screens too.
ReplyDeleteOh, and for the guy with the "This post is probably where I got the most useful information..." posting seriously relies one obscure web blog as his main source of reliable information, I pity his clients. I hate to be the one to wake you from your dream-like state buddy, but a 3-second search on Google will find you more information about the real state of automated testing than you're gonna find here...unless you've already made up your mind and are just looking for someone to support it. If that's the case, then by all means pile on. In reality, it'd be like asking your local Ford dealer if Toyota trucks are any good. Nope, not gonna be any bias there. LOL...
Unfortunately Alan here has hit it on the head.
ReplyDeleteI have mainly worked for some of the best companies out there (who shall remain nameless), and yes for fortune 500 types (and designed some test harnesses that actually work), and operated in several startups (where I have avoided QTP like the plague and achieved good results with OSS).
Many, many of those (fortune 500) have purchased and used tool sets like QTP and including QTP.
Following which they either:
Don't use them
Fail while attempting to use them
Fail without knowing they are failing - low coverage, huge sets of metrics that make it look like stuff is getting done when it's not / etc.
QTP is mostly a way to burn your company's cash while making it look like you're doing 'something'.
In rare cases the people using it actually know how to use it and get good results. But this is really pretty rare in my experience. It's all smoke, mirrors, and lies mostly.
And as an addendum and response to the post above: college interns cannot 'pick this up in an hour'.
ReplyDeleteDesigning and implementing reliable, scalable, maintainable, dare I mention 'data independent'?? test code is a task for seasoned professionals and not for kids regardless of the tools used.
As a HP employee I am going to defend my company. I am experienced in QC and I am just getting introduced recently to QTP so my comments will focus on my knowledge on QC.
ReplyDeleteQC has been profitable software for HP for many years now. If it weren't, HP would have taken it off the market a long time ago. In a company this big all business is reviewed quarterly and if something is no longer being profitable you better believe it will better have a damn good reason why we should keep investing on it. Granted, profitable doesn't mean it is being used. But the thing with this tool is that, due to price (which I do agree is expensive and will not work for the small new software company with 5 employees and is just barely making it) is that every time HP makes a business with a company, many times it is included in a bundle. That is, HP will also provide consulting expertise, provide the servers, and guess what else? Of course, QC! Rarely will a customer just come to HP and say, “Hi, Could I order 1 HP QC to go, please?” And in these sometimes millions of dollars negotiations, a few hundred thousand represents only a fraction of the price. And when one of this big negotiations occur, hundreds if not thousands of users will be using QC. This is where QC really shines. Try managing a project with 200 users all trying to log your defects into your “fancy version of Microsoft Word”. Try managing 100 of these projects on your “fancy version of Microsoft Word”. So I do agree with your statement that many times testers and designers are forced to use these tools. But that should not disqualify them as good tools. Is QC the best testing tool right now out there? Probably not. Could QC be improved? You bet! Furthermore, it is not HP’s fault that customers do not take full advantage of all the capabilities the tool offers. For instance, in my opinion one of the coolest features of QC is the ability to have test cases libraries, so that you do not have re-invent the wheel with every project writing new test cases. Unfortunately, even our biggest clients lose thousands in dollars in man hour rework by not using this capability for which they have already paid. Crazy, isn’t it? Same goes for requirements module included in QC for instance. Furthermore, also note that HP is also investing in new testing tools that offer many cool features and adapt to new testing environment like HP Agile Manager. Check it out, they have a free trial on the website. Perhaps that one may help change your mind regarding HP software testing solutions.
So, to answer your initial questions, “Does anyone actually use Quick Test Pro etc.?” Yes, thousands of persons use QC. Perhaps not in many small/medium companies, but if you look around in large companies you will see how it is used everyday.
Thanks anonymous,
DeleteHP should be period that one of their employees is prepared to defend the company.
You have certainly added to my understanding of the dynamics of Quality Centre.
In truth QC is not an automated testing till so it is wrong of me to lump it in the same bracket. I do so because I find its use is often symptomatic of the issues I describe.
Thanks again.
allan
Allan - I just saw this post as a link from one of your other posts, so I am late to the game. I come from a hardware test automation background so I am probably not a typical tester example you are citing. About ten years ago, I was called in to help a team testing a Java enterprise application, where the middle tier and client communication tier were not accessible, so they had to do GUI testing. They already had purchased a QC/QTP suite license with LoadRunner to do the testing (before they had a plan). I ended up using QTP and threw away the 90% that tied to the mothership (QC). I used spreadsheets for recording results and wrote a GUI search algorithm to make the automation robust to changes. Then used that same algorithm to test a web application using QTP and then another Java enterprise application using TestComplete (much less expensive). In each case, I found it nearly impossible to pass on my accomplishments to others - either the testers didn't have the the proper coding skills or the developers were only interested in unit test automation.
ReplyDeleteBTW, instead of using loadrunner I teamed with a developer and set up the open source Grinder load test tools, which worked superbly in the Java enterprise environment.
As a result of these experiences, I now believe that the best role of automation is to help the exploratory testers as much as possible, but the test skills based on exploration should always come first.