Testing Concepts and Test Automation

Software Quality Revolution

Software Testing is a process that consists of all test life cycle activities like static and dynamic testing concerned with planning, preparation and evaluation of software products to determine that the software products satisfy customers requirements and are fit for customers use. Software Testing is done to find software defects or failures in advance.

Lean principle
A systematic approach to identifying and eliminating waste through continuous improvement, flowing the product at the pull of the customer in pursuit of perfection.

The Quality Revolution

It illustrate the activity sequence:

Setting goals Assigning them to measurable milestones Assessing the progress against the milestones Take action to improve the process in the next cycle

The Objectives of Testing

It does work
It does not work

Reduce the risk of failures

Reduce the cost of testing

The Concept of Complete Testing

Complete or exhaustive testing means There are no undisclosed faults at the end of test phase

Complete testing is near impossible for most of the system The domain of possible inputs of a program is too large Valid inputs Invalid inputs The design issues may be too complex to completely test It may not be possible to create all possible execution environments of the system

Failure, Error, Fault and Defect

Failure A failure is said to occur whenever the external behavior of a system does not conform to that prescribed in the system specification Error An error is a state of the system. An error state could lead to a failure in the absence of any corrective action by the system Fault A fault is the adjudged cause of an error Defect It is synonymous of fault It a.k.a. bug

Testing Level

Unit testing Individual program units, such as procedure, methods in isolation Integration testing Modules are assembled to construct larger subsystem and tested System testing Includes wide spectrum of testing such as functionality, and load Acceptance testing Customers expectations from the system Two types of acceptance testing UAT BAT UAT: System satisfies the contractual acceptance criteria BAT: System will eventually pass the user acceptance test

V model

Test Automation

But, Is This Really Automation?

Analyze

product Design test Run test 1st time Evaluate results Report 1st bug Save code Save result Document test
Re-run

-----------

human human human human human human human human MACHINE

MACHINE

the test
result
(plus

Evaluate

human is needed if theres any mismatch) human

Maintain

result --

Automated Regression Pros and Cons

Advantages

Disadvantages
Breaks Tests Pays

Dominant automation paradigm Conceptually simple

easily (GUI based)

are expensive off late to failure because:

Straightforward
Same approach for all tests Fast implementation Variations are easy Repeatable tests

Prone

difficult financing,
architectural, and maintenance issues
Low

power even when successful (finds few defects)

GUI Automation is Expensive

Estimates run from 3-5 times the time to create and manually execute a test case.
You usually have to increase the testing staff in order to generate automated tests. Otherwise, how will you achieve the same breadth of testing? Your most technically skilled staff are tied up in automation Automation can delay testing, adding even more cost (albeit hidden cost.)

Maintaining GUI Automation

GUI test tools must be tuned to the product and the environment GUI changes break the tests
May need to wait for GUI stabilization Most early test failures are due to cosmetic changes False alarms are expensive We must investigate every reported anomaly We have to fix or throw away the test when we find a test or tool problem Maintainability is a key issue because our main payback is usually in the next release, not this one.

Common Mistakes about Test Automation

Dont expect to be more productive over the short term.
The reality is that most of the benefits from automation dont happen until the second release. It takes 3 to 10+ times the effort to create an automated test than to just manually do the test. Apparent productivity drops at least 66% and possibly over 90%. Additional effort is required to create and administer automated test tools.

Think About

Automation is software development. Regression automation is expensive and can be inefficient. Automation need not be regression--you can run new tests instead of old ones. Maintainability is essential. Design to your requirements. Set management expectations with care.

Issues Faced in a Automated Test

What is being tested? How is the test set up? Where are the inputs coming from?

What is being checked?

Where are the expected results? How do you know pass or fail?

TestComplete

TestComplete Properties

TestComplete Explorer

TestComplete Run

TestComplete Report - Details

TestComplete Report - Details

Thank You
Ganesh B.