process improvment

Exploratory Testing – Good And Bad Sides

Let’s check wikipedia citation “Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The quality of the testing is dependent on the tester’s skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be”.
I will count the good and bad sides of exploratory testing which I met in my quality assurance experience. At the beginning, the question “Is it worth it?” I answer “Absolutely!” All techniques and methodologies have the shaded areas, the key is to get the best value and reduce the drawbacks.

Good Sides

Checks application usability
This of course depends on how the tester is sensitive to usability. Usually, however, something that is understandable for the developer and the robot, it is not obvious to the tester. Because exploratory tests are treated as a whole, it is easier to see the wrong assumptions in cross-section of multiple modules or functions.

Helpful with a lack of documentation, requirements, test cases, etc
There are situations when we get application to test without any documentation, test cases or automation scripts. Natural way is, of course, writing the test scenarios first, however, I recommend to pre-order exploratory tests. Soon we will have the information about the quality level of the product – our benchmark. Testers will have a good basis to start writing the missing parts. So this kind of testing allows you to get temporary salvation and able to maintain the expected quality in short term.

Find holes in requirements
Exploratory tester usually report many errors caused by wrong requirements or documentation. What is interesting is such errors are usually reported as critical. As mentioned earlier, deduction across whole applications have no small importance here. Exploratory testing can upset upside down some of the assumptions.

Bad Sides

We do not need test scenarios, unit tests, test automation, etc. ! We have skilled testers !
The situation is often encountered in smaller firms where there is no dedicated quality assurance teams. Tester is a programmer, a tester is a business owner, a secretary, all strive to perform exploratory tests. At the end of their testing, it becomes a set of performed routinely activities. For larger companies, we also deal with the syndrome, “Our application is free of errors!”.

Poor detection of minor issues
Exploratory tester is focused on finding gaps in mainstream business process covered by tested application. But of course there are also deviations in the other side, in example : focusing on the type of error – whether it is possible to enter a negative value in the field, which should has only positive values. Fields validation is a role of automated or unit testing.

Exploratory testers can get into a routine
Described methodology is based on the deduction, which degrades when is constantly exposed to the same experience. Tester which is extensively used in this way, often becomes an automation robot that uses a memorized test script. As team leaders we have seen this phenomenon in advance. The easiest way to counteract this situation is to apply the testers and test applications rotation.

Summary
I think that exploratory testing is a “must have” methodology in your testing process. In larger teams it may be a group of dedicated testers, the smaller one or two people with extensive experience. Here are a few links to interesting posts on this topic.

Links
Explaining exploratory testing
Exploratory testing versus test cases
Exploratory testing vs scripted testing
Automated Exploratory Testing

Discussion

8 comments for “Exploratory Testing – Good And Bad Sides”

  1. I can’t quite agree with Exploratory Testing (ET) being a methodology. It’s an approach to testing. According to Alistair Cockburn a methodology includes Roles, Skills, Teams, Tools, Techniques, Activities, Work Products, Standards, Quality, and Values. A methodology is an agreement of how multiple people will work together. (see http://alistair.cockburn.us/Methodology+per+project)

    Second, the bad sides you mention appear due to other facts. “We do not need test scenarios, unit tests, test automation, etc. ! We have skilled testers !” is the misbelief, that exploratory testing alone is enough. Neither is automated checking. The combination of both – when applied well with well skilled professionals – leads to a well-balanced serving.

    “Poor detection of minor issues” relies heavily on the skills of the testers. Exploratory Testing may be taught and skill can be trained.

    “Exploratory testers can get into a routine” is a misconception. Session based test management can mitigate this effect to a great deal. Setting up varying charters within these sessions with appropriate debriefings avoids the routine part. In this case I would claim the test manager is misinformed about testing.

    Posted by Markus Gaertner | January 26, 2010, 1:56 pm
  2. Hmm, I have some differences of opinion with most of the things you have writen. What you have put up is not a well thought out or researched piece of writing. Allow me to elaborate.

    Checks application usability
    Hmm maybe. I can think of a lot of exploratory testing that I have done that has nothing to do with usability however. Exploratory testing allows you free rein to notice and report things like usability issues even when that is not the focus of your testing. In that you are correct, but to say that checking of usability is an inherent quality of exploratory testing would be incorrect.


    …Usually, however, something that is understandable for the developer and the robot, it is not obvious to the tester. Because exploratory tests are treated as a whole, it is easier to see the wrong assumptions in cross-section of multiple modules or functions.

    Not really sure what you mean by this. If you are suggesting that developers think at the unit level and robots don’t think at all, there may be some truth to that, (depending on the developer of course), but this has nothing to do with exploratory testing. Testing of multiple modules and end-to-end can be done in a scripted fashion to good effect, so again I wouldn’t say this is a quality inherent to exploratory.

    Helpful with a lack of documentation, requirements, test cases, etc
    There are situations when we get application to test without any documentation, test cases or automation scripts. Natural way is, of course, writing the test scenarios first, however, I recommend to pre-order exploratory tests. Soon we will have the information about the quality level of the product – our benchmark.

    Okay, you’re on the right track here (though I’d debate test case writing being more natural), but…

    Testers will have a good basis to start writing the missing parts. So this kind of testing allows you to get temporary salvation and able to maintain the expected quality in short term.

    The first part of this sentence may true. You may find that there are areas that you want to write test cases for, or you may decide that that writing a number of new test charters could be the way to go. What I have no idea what you mean about temporary salvation or maintaining expected quality. Maybe you’re talking about having a test-case-counting nazi manager in which case, fair enough (though that is a separate problem), but what do you mean about maintaining expected quality? What can a tester do to maintain quality?

    Find holes in requirements
    Exploratory tester usually report many errors caused by wrong requirements or documentation. What is interesting is such errors are usually reported as critical. As mentioned earlier, deduction across whole applications have no small importance here. Exploratory testing can upset upside down some of the assumptions.

    What do you mean by ‘usually reported as critical’? I don’t get what you mean by this as it relates to exploratory testing. As for the rest, I think I understand what you’re saying and I think I agree, but you’re not being clear here. Can you clarify?

    Okay, I hope you’re still with me because it’s your cited down sides that I really have issues with.


    We do not need test scenarios, unit tests, test automation, etc. ! We have skilled testers !

    The situation is often encountered in smaller firms where there is no dedicated quality assurance teams. Tester is a programmer, a tester is a business owner, a secretary, all strive to perform exploratory tests. At the end of their testing, it becomes a set of performed routinely activities. For larger companies, we also deal with the syndrome, “Our application is free of errors!”.

    No. You are either amalgamating two separate issues, or you are mistaking exploratory testing for poor testing. What your paragraph describes is not what your heading suggests. The paragraph describes sloppy or unskilled testing. Your heading implies hubris that I might expect from an unskilled tester, but no one that I know who understands exploratory testing would suggest that unit testing or automation was not necessary due to the existence of ET.

    Poor detection of minor issues
    Exploratory tester is focused on finding gaps in mainstream business process covered by tested application. But of course there are also deviations in the other side, in example : focusing on the type of error – whether it is possible to enter a negative value in the field, which should has only positive values. Fields validation is a role of automated or unit testing.

    Can you cite any reference to back up your first sentence? Why do you think that exploratory testing would not be helpful in boundary / edge case testing? Does the name exploratory not lend itself to the investigation of these very things? Fields validation is a role of auto or unit testing? (and not any other kind?) – No. You are mistaken. This is very much a domain of ET and I would expect it to be the domain of any skilled tester whatever approach to testing they use.

    Exploratory testers can get into a routine
    Described methodology is based on the deduction, which degrades when is constantly exposed to the same experience. Tester which is extensively used in this way, often becomes an automation robot that uses a memorized test script. As team leaders we have seen this phenomenon in advance. The easiest way to counteract this situation is to apply the testers and test applications rotation.

    Sorry, did you perhaps mean to write scripted testers here? I cannot make sense of what you have written. What you are describing is a person who runs and repeats a script. This is the antithesis of exploratory testing. Good exploratory testing is about using what you learn to inform what you do next. By its definition, good ET will not get into a rut or become robotic because the tester, even if working to a charter or directive, has license to deviate as they see fit as the situation dictates.
    What you are describing is repetition fatigue, which could happen using any approach if you test the same thing long enough. This is not a drawback of ET, it is poor test management.

    I don’t believe you are setting out to malign ET as you describe it as a necessary part of testing, but you appear to be terribly misinformed about what ET is. You talk about it as a methodology, but the points you make point to it being a technique. I suggest you spend some time reading material by James Bach, Michael Bolton and Cem Kaner to better acquaint yourself with ET.

    Posted by Ben Kelly | January 26, 2010, 2:13 pm
  3. Hi Markus,

    Many thanks for the comment. Writing this post I hope to start a little discussion.

    You have right that ET itself is not a methodology, but many quality departments are based on it as a whole. They build some kind of methodology on ET concept.

    ET alone is not enough ! When you need, in example tool to automate the tests, you download 10 items then you run ET on each of them. We do ET all the time and, in my opinion, it is sufficient for 0 step checking.

    Of course you’re right, badly run ET may lead to tester routine. I wrote this post to prepare readers for such events 🙂

    Posted by Marcin Zręda | January 26, 2010, 2:24 pm
  4. Hi Ben

    Well, we have lift off that a substantive discussion 🙂

    First: Usability is not a ET goal ! My experience shows that many errors reported during these tests is related to usability. ET helps us to improve usability but somehow in the second thread 🙂

    Second: Lack of docs and temporary salvation 🙂 I have several accidents when I had to do some kind of fast checking third party component or application and I that case ET was a salvation. When the docs and reqs are not clear or not complete, testers can not check whether current solution have issues, idea is wrong or documentation is not complete. Again, that are my experiences 🙂

    Third: “We do not need test scenarios…” – I see that I was not understood 🙂 This sentence have false assumptions, who say it do not understand the role of ET or use only those tests and do not know about others!

    In conclusion, I would say that the thesis contained in this post are theses which were to trigger a discussion. I am a fan of ET but it frightens me to treat them as a sufficient or only approach. There are many misunderstandings in this area. The good and bad sides mentioned here are no objections to a methodology or technique, talk about perception of ET in different way (not right way).

    Posted by Marcin Zręda | January 26, 2010, 3:07 pm
  5. Where I come from, this is called trolling and it’s generally not considered a cool thing to do.
    What you have done is present a number of fallacies as facts. If your intention was to start a discussion based on your own experiences, why did you not state that explicitly? You use far too many hand-wavy generalisations. Be specific about your experiences, your concerns. Differentiate your opinions from fact.

    You have given us no indication as to what your experience of ET is or how you’ve used it, we simply see a number of things that you assert based on what that experience is. You haven’t juxtaposed that with the experience of other testers, so what you have are conclusions drawn from an undescribed experiment with a sample size of one.

    Where you say it frightens you to treat ET as the only approach – do you have experience with someone advocating ET describing it as the only way? I do not.

    If you want to have a discussion about your experiences with ET, I suggest that in future you present your material as exactly that. I would also suggest that you compare your experiences with experienced advocates of ET to see where your experience differs, so at the very least you can say ‘this person says X. In my experience I have seen Y’.

    I didn’t comment here for the sake of having a discussion. I commented in order to correct what I see as misinformation about exploratory testing.

    There is already enough online garbage to wade through when learn about testing. Please strive to be a voice of reason, learning and teaching. This piece is not that.

    Posted by Ben Kelly | January 27, 2010, 12:39 am
  6. Hi Ben,

    I’ve updated the post with a little explanation at the start.

    Thanks for active keeping hand on the pulse

    Posted by Marcin Zręda | January 27, 2010, 9:26 am
  7. […] I’ve kept notes on what has worked and what hasn’t, and a recent post by Marcin Zręda (Exploratory Testing – Good And Bad Sides) got me thinking about the notes I’ve […]

    Posted by Dwimordene » Blog Archive » Exploratory Testing | January 30, 2010, 8:05 pm
  8. Again, the ET thugs come out.. whenever someone challenges the ET manifesto the ET clowns come out; they attack the criticism with silly support – reading about made my head hurt with the circular arguments.

    It’s sad, but your post is spot on, it’s a philosophy, and you tried to outline the good and the bad and you were attacked and then accused of being a troll – in which you defended your remarks.

    sad

    Posted by Anon | April 24, 2010, 10:41 pm

Post a comment