Documente Academic
Documente Profesional
Documente Cultură
Texas A&M University Department of Computer Science CPSC 606: SOFTWARE ENGINEERING Presentation 2 Sameh S. Sharkawi
Outline
Introduction
What is MBT Motivation behind MBT
How MBT Works Benefits of MBT Problems of MBT Current State of MBT in Industry Software Tools for MBT Conclusion References
Definition Model-based testing refers to software testing where test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT) .
*
* Wikipedia
Testing is expensive
30-50% of development costs Down time Maintenance costs Rework Law suits
Brokerage operations $6,450,000 Credit card authorization $2,600,000 Ebay (1 outage 22 hours) $225,000 Amazon.com $180,000 Package shipping services $150,000 Home shopping channel $113,000 Catalog sales center $90,000 Airline reservation center $89,000 Cellular service activation $41,000 On-line network fees $25,000 ATM service fees $14,000
The model is usually an abstract, partial presentation of the system under test's desired behavior. The test cases derived from this model are functional tests on the same level of abstraction as the model. These test cases are collectively known as the
The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. This is done by mapping the abstract test cases to concrete test cases suitable for execution.
Implementation
Benefits of MBT
Benefits of MBT
Involves testers early in the development process Teams testers with developers Forces testability into product design Finds design and specification bugs - before code exists The model is the test plan - and is easily maintained Automated test suite generation Coverage is guaranteed - increases testing thoroughness Zero test suite maintenance costs Finds code and interface bugs Includes a framework for the testing of distributed applications Reduces test execution costs
Machines
41 500 ($50 000 USD) per year Works 40 to 50 hours per week
250 ($300 USD) per year to buy and operate Can work 100 hours per week
react.cs.uni-sb.de/mbt2006/talks/mbt4masses.pdf
Problems of MBT
Problems of MBT
Process shift
Personnel shift
IBM Research
Most projects were done by PhD Holders Very Little was successful (Just in the Lab, never actually went into market) Too Complicated to use and analyze
Microsoft Research
The Spec# programming system is a new attempt at a more cost effective way to develop and maintain highquality software
Adds pre/post conditions, contracts Adds high-level data types with convenient notations Adds logical quantifiers like FORALL and EXISTS
You can run it like a C# program It can call framework code NEW You can explore it (run all of its possible behavior) which is how we do MBT with Spec#
A model exploration and testing tool for .NET: Authoring of models in Word Model exploration (generating FSM) FSM visualization FSM traversal and test suite generation Automatic implementation binding Online test-suite execution and offline test-suite code generation
Model: Calculator
bool Running = false; bool Scientific = false;
System State
[Action] void SetRunning(bool newRunning) requires Running != newRunning; { Running = newRunning; } [Action] void SetScientific(bool newScientific) requires Running; requires Scientific != newScientific; { Scientific = newScientific; }
Spec Explorer (UW-MSR Summer Institute '04)
Spec#: Pre-Condition State Update describes when action is enabled Tells Spec Explorer this is an action. Action invocations do appear in tests.
Conformiq Test Generator is a model-based testing tool using UML state charts that represent testing strategies Leirios Test Generator is a model-based testing tool that generates tests automatically from deterministic system specifications Reactis Tester is another model-based testing tool that focuses on control systems TGV is a tool for the generation of conformance test suites for protocols TorX is also a prototype testing tool for conformance testing of reactive software Lurette is an automated testing tool of reactive programs written in Lustre AsmL Test Tool can generate tests directly from an AsmL model AutoFocus (in german) is a graphical tool for developing and modeling distributed systems with integrated testing facilities
AsmL is the Abstract State Machine Language. It is an executable specification language based on the theory of Abstract State Machines. The current version, AsmL 2 (AsmL for Microsoft .NET), is embedded into Microsoft Word and Microsoft Visual Studio.NET. It uses XML and Word for literate specifications. It is fully interoperable with other .NET languages. AsmL generates .NET assemblies which can either be executed from the command line, linked with other .NET assemblies, or packaged as COM components.
Conclusion
Conclusion
Questions
How do we guarantee that the generated expected outputs and inputs are enough and cover all the cases? How to make for the errors and exceptions that may occur due to running the program on different environments and different OSs? How is automated generation of inputs and outputs work when an actual human interaction is required as part of the system testing? Does MBT cover stress testing? Does MBT cover how user friendly the system is or it only covers functionality?
References
References
http://en.wikipedia.org/wiki/Model-based_testing http://blogs.msdn.com/nihitk/articles/144664.aspx http://research.microsoft.com/specsharp/ http://react.cs.unisb.de/mbt2006/talks/mbt4masses.pdf http://react.cs.unisb.de/mbt2006/talks/ModelBasedTestingSoberEval uation.pdf http://aetgweb.argreenhouse.com/papers/1999icse.pdf Colin Campbell, Wolfgang Grieskamp, Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Veanes Spec Explorer: An Integrated Environment for Model-Based Testing. Foundations of Software Engineering Microsoft Research, Redmond