Test Case Passed! Who Cares!!! - Pradeep Soundararajan

Hello!
I am Pradeep Soundararajan from Bangalore, India
Copyrights Pradeep Soundararajan http://testertested.blogspot.com pradeep.srajan@gmail.com
I test, manage, coach, consult, speak, write on software testing.
Here is a story of how I struggled to find bugs a few years ago, which I thought might be of help to you.
No, this is not a story without a name, so here is the name
The snake in the monkeys shadow
Test case Passed! Who cares?
oOPS! I think that wasnt a title that makes sense to this context. OK, Let me rename it
The title will appear above the above text in 45 seconds !
To start the story, here is an expected result of a test case FR_1223 that I executed! The application should handle the input and should not hang or crash the application.
I executed that test and passed the test because the application did handle the input and did not crash/hang the application.
Are you curious to know what happened once I passed the test?
The customer complained that although the input was handled, the application took 15 seconds to perform the operation.
but hey, the Spec didnt say that?
Well the test passed but why did I not catch the bug reported by the customer?
I then added another point to the expected result:

Expected Result of test case FR_1223:
The application should not take more than 5 seconds to handle it.
The application should handle the input and should not hang or crash the application.
I executed the test again in the next release and the test passed!
Yet again, the customer came back logging another issue: While the application now takes 5 seconds to handle, it fails to process any input once I close and reopen the application after an abrupt closure during those 5 seconds.
I then updated the Expected Result of test case FR_1223 to:
The application should handle any abrupt closure or there should be a proper warning or error message during that operation.
The application should handle the input and should not hang or crash the application. The application should not take more than 5 seconds to handle it.
And in the next build I executed the test and it passed!
Our customer came back logging another issue: While the application now handles abrupt closure by warning the user of loss of data, the database elements are corrupted when the operation is iterated 5 times.
I then updated the Expected Result of test case FR_1223 to:
When such an abrupt closure is carried out multiple times
the database should not be corrupted.

The application should handle the input and should not hang or crash the application. The application should not take more than 5 seconds to handle it. The application should handle any abrupt closure or there should be a proper warning or error message during that operation.
Our customer was quite happy that he could not find anymore issues with respect to that feature and most of his expectations were met. Wow! The story appeared to end. I was simply wrong!
Our customer then passed the release to their customer. Our customer came back saying their customer isnt happy because the Submit button that performs an action of processing the input is too small to be noticed for new users.
I then updated the expected result to:

All radio buttons associated with this operation should be of size X by X (as communicated by the customer in e-mail dated October 22)
The application should handle the input and should not hang or crash the application. The application should not take more than 5 seconds to handle it. The application should handle any abrupt closure or there should be a proper warning or error message during that operation. When such an abrupt closure is carried out multiple times the database should not be corrupted.
And in the next build I executed the test and it passed!
Yet again, our customers customer, reported an issue: It appears that when this operation is being processed in IE, the browser shows Not Responding and then recovers.
I then updated the expected result to:

The application should process the operation in a same way across multiple browsers ( IE, Firefox, Safari, Web Monkey, Netscape )
The application should handle the input and should not hang or crash the application. The application should not take more than 5 seconds to handle it. The application should handle any abrupt closure or there should be a proper warning or error message during that operation. When such an abrupt closure is carried out multiple times the database should not be corrupted. All radio buttons associated with this operation should be of size X by X ( as communicated by the customer in e-mail dated October 22 )
The story went on for a while and today when I look back, I realize I learned a few important lessons in testing.
I am wondering if I should share the lessons with everybody instead of people whom I choose to share?
OK, so enter the password that I might have sent you to read the lessons, else you dont get to read them and I am sorry about it.
Password:
Check
Wait a minute! I changed my mind.
You can read them without entering the password and here it goes
Lesson Number Minus Three

The specification could have been met long back but it is not the specification that approves the product nor the ones who wrote the specification but it is *humans*- who arent completely bothered if specification is met but bother if it meets their expectations without bothering if their expectation was documented in the specification or not.
Lesson Number Minus Fou
If you think your customer learns new things every day as you do, then you must be aware that their expectations keep changing as they learn, just like how it is happening for you. Do you expect the same salary once you have learned that your contribution to a project that you work has grown humongous?
If you expect the same you deserve the same
Lesson Number Minus Five
If all expectations were to go in to a column of test case whose heading is Expected Result then each element of the column might be of several thousand words and hundred points as compared to what we most commonly get to see even if we were to write those test cases. It might take about an hour just to read such a test, isnt that funny?
Lesson Number PLUS six ( Were you expecting a MINUS six? )
When I practiced using heuristics* and oracles** (not the database) while exploratory testing a product I found that I loaded my brain with the list of oracles and heuristics and then shot every test with as many expectations I could think of and I was able to find not just one bug with every test I did but many.
Heuristic* A fallible method of solving a problem Oracle** - A principle of mechanism by which we (humans and not the process) identifies a problem
Lesson number minus seven-teen

I recommend other testers to not use the Heuristics and Oracles approach to their testing because I am a little selfish about being one of the few testers who finds many important problems, quickly.
I also dont recommend you to go through Rapid Software Testing material that helps you in learning a lot more about Oracles and Heuristics which can be found at www.satisfice.com/rst.pdf
Lesson number eight
If I had all lessons spoon fed to me, I would have become the laziest tester in the world and I think I did not become that because I realized the one of the solid ways to do a good testing, is by practicing doing good testing.
Lesson number Aight!

When I execute tests that have words like handle the input looks reasonable correct results are displayed My opinion about those words might be different from those of my customers and even other testers.
Lesson contribution from Ben Simo www.questioningsoftware.com

Lesson number 8 ( not the word eight ) A different lesson dont worry about that!
I think we learned so many things with the story and I guess we should have also learned that it is humans who solve problems and not the process they make, which many think takes care of testing. A process is a heuristic and can never be a best practice!
The first time I heard Heuristic and Oracles

I had the same feeling as you might be having now Greek and Latin but I asked myself a question, If I can do a lot better testing by learning that Greek and Latin, why not I learn it? and here I am writing stories and articles about them after exploring the basic power of those two words.
Thank you for your time!
This presentation is inspired by Michael Boltons Emotions and Oracles Tutorial and the creative (in my opinion) style of presenting was inspired by Ben Simos FAILURE presentation (that is not public yet) and from the titles of the movie Monty Python and the Holy Grail.
I admit that I didnt do anything more creative than those!
Some of you might have thought, after going through this presentation (so far) that I am trying to say, I no longer struggle to find bugs because I exploit the power of Heuristics and Oracles.
I am glad to say to those who thought that way, I think you are wrong, I still do struggle but I might not struggle on the same problems that I struggled last year or maybe yesterday
For more such stuff that might interest / bore you like this keep visiting my blog http://testertested.blogspot.com
Lessons are not for free they are mixed with some marketing, so bare with it
The END!
http://testertested.blogspot.com pradeep.srajan@gmail.com "Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton

Test Case Passed! Who Cares!!! - Pradeep Soundararajan

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Test Case Passed! Who Cares!!! - Pradeep Soundararajan

Încărcat de

Drepturi de autor:

Formate disponibile

Hello!

I am Pradeep Soundararajan from Bangalore, India

Copyrights Pradeep Soundararajan http://testertested.blogspot.com pradeep.srajan@gmail.com

I test, manage, coach, consult, speak, write on software testing.

Copyrights Pradeep Soundararajan http://testertested.blogspot.com pradeep.srajan@gmail.com

Copyrights Pradeep Soundararajan http://testertested.blogspot.com pradeep.srajan@gmail.com

No, this is not a story without a name, so here is the name

The snake in the monkeys shadow

Copyrights Pradeep Soundararajan http://testertested.blogspot.com pradeep.srajan@gmail.com