1 Introduction

1.1 General The main purpose of this project is to automate the web client which is CommVaults Unified Data Management that uses the Windows Server platform and the Microsoft FAST Indexing engine to provide an E-Discovery solution that ensures microscopic data management for the heterogeneous environment. The data that is being populated in this web client is of various types like Exchange, SharePoint, Office Communications Server, Windows, UNIX, Linux, MAC, NetWare and NDMP(Network Data Management Protocol) locations stored on various hardware. So Automation framework is used to automate a battery of test scenarios that takes the input as parameters and a static dataset and produces reports and logs.

1.2 Motivation The traditional testing i.e. manual testing is intended to check the application by putting it through a defined series of steps and by analyzing the result after each step. But the manual testing is cumbersome and slow. So the steps are organized in such a way that it can exercise all the functionality of the application as well as reiterate the scenarios that have identified with the problems. In order to do so we have designed an automation framework that is based upon the xUnit architecture for unit testing framework. It comprises of JUnit framework, Selenium RC, Maven and log4j that uses Java to write the code.

This framework finds its motivation in automating the black-box/system testing (which encompasses functional testing) of the product under test also known as AUT (Application under test).

This Framework can be particularly useful in the regression testing stage. Regression testing involves executing a pre-defined battery of tests against successive builds of the system under test to verify that bugs are being fixed and features / functions that were working in the previous build haven't been broken. Regression testing is an essential part of testing, but is very repetitive and can become tedious when manually executed build after build. The real benefits of using this framework can be reaped when applied for periodic regression testing.

The UI acceptance tests can be written using this framework which enables the engineer to run the complete application as a single test from IDE (like Eclipse or Net beans) or using command prompt via Maven commands. So it can be used for acceptance testing also.

1.3 Aims and Objective Learn about the technologies and applications of browser based automation framework Understand the advantages and the drawbacks of automation rather than manual testing Identify the automation framework that will have high tolerance with the changes made Reduction in test cycle time and effort, which in turn should result in accelerated time-tomarket and reduction in the total cost of testing Reduction of manual intervention Improvement in test case execution repeatability Ability to provide parallelism in testing Simulation of realistic load generation (load ramping) Capability to carry out result analysis and logging for all tests results Achieving the maximum degree of test coverage Support for product evolution, i.e. the testing framework should support future extensions to the test hardware and products. 1.4 Related Work This Framework uses JUnit framework. JUnit framework is the extension and revamping of xUnit architecture that is used for agile testing. The xUnit all have the same basic architecture. This project describes the xUnit, using JUnit as the reference example, since it is the most widely used of the xUnit. The other xUnit vary in their implementation details, but follow the same pattern and generally contain the same key classes and concepts. The key classes are TestCase, TestRunner, TestFixture, TestSuite, and TestResult. The main advantage of xUnit frameworks is that they provide an automated solution with no need to write the same tests many times, and no need to remember what should be the result of each test.

All xUnit frameworks share the following basic component architecture, with some varied implementation details. Test case This is the most elemental class. All unit tests are inherited from here. Test fixtures A test fixture (also known as a test context) is the set of preconditions or state needed to run a test. The developer should set up a known good state before the tests, and return to the original state after the tests. Test suites A test suite is a set of tests that all share the same fixture. The order of the tests shouldn't matter. Test execution The execution of an individual unit test proceeds as follows: setup (); /* First, we should prepare our 'world' to make an isolated environment for testing */ ... /* Body of test - Here we make all the tests */ ... teardown (); // In the end, whether succeed or fail we should clean up the environment. The setup () and teardown () methods serve to initialize and clean up test fixtures. There are various variants of xUnit architecture like JUnit, NUnit, CPPUnit, PyUnit, XMLUnit etc. CppUnit is a part of JUnit to C++. CPPUnit implementation details differ from JUnit as a result of design choices by CppUnit's developers and language differences between Java and C++. CppUnit's implementation makes full use of advanced C++ language features, including templates, abstract classes, nested classes, and the Standard Template Library (STL) It also makes extensive use of C macros, which some consider inelegant and error-prone, but are definitely useful here. CppUnit is designed to be thread-safe. NUnit is a unit test framework for the Microsoft .NET architecture. Conceptually, it follows the xUnit model, serving as a foundation for building unit test classes and methods. It is implemented in C#, but supports writing unit tests in any .NET language, including C#, J#,
3

Managed C++, and Microsoft Visual Basic .NET (VB.NET). NUnit defines tests using C# attributes rather than object inheritance, so the details of its software architecture differ significantly from JUnit. PyUnit brings xUnit to Python. Python is an interpreted, interactive, object-oriented programming language, widely used for many different kinds of software development. It follows the generic xUnit model closely. XMLUnit provides useful support for unit testing of XML content. It is an extension to JUnit and NUnit, rather than a standalone framework. Development and testing of software that generates XML content are common activities. Writing such tests is difficult with generic unit test frameworks. XML tests often compare expected XML content to actual XML content generated by the software that is being tested. Such tests can be written by doing a string comparison between the expected and actual XML content strings. However, what if the actual XML content contains a return character for readability, or puts XML attributes in opposite order than they appear in the expected XML string? Taken as XML documents, the two XML strings have the same contents, but an exact string comparison will fail. 1.5 System Requirements 1.5.1 S/W Requirements Operating systems: Windows (all variant including servers), Linux, UNIX. The platform should support JVM (Java Virtual Machine). RAM Required: 4 GB Physical memory. Java Version: Latest JDK(Java Development Kit) Maven (3 onwards). IIS (Internet Information Server) is must to run the web client. Tomcat- latest version to run services.

1.5.2 H/W Requirements There is no hardware requirement.

1.6 Report Organization The report is organized according to IEEE standard. In Section 3 we will talk about Analysis and design with the detailed studies of the various components that are used in the project. It will talk about the various attributes of the framework in reference of requirements. Section 3 also elaborately discusses about the assumption and dependencies of the framework. It discusses about the type of user and environment. Section 4 discusses the implementation, tools that are used in the implementation and user interface design. It also shows the UI in the form of screen shots. Section 5 discusses about the testing aspect of the framework. Section 6 discusses the results and future enhancements.

2 Overview of the Proposed System

2.1 Introduction of problem and its related concepts Manual testing procedures often fail to reveal all the defects that they should, because testers often have selective or incomplete experience and knowledge of the application and its domain. The test coverage of test cases is invariably not known and if tracked, is usually inadequate.

Invariably, there is a lack of resource level planning in a manual testing scenario and critical test hardware and software resources required for testing are not being used optimally. There is also a possibility of skipping some of the key changes that are included in the built due to gradual development because these changes are not very apparent.

The organization chooses the automation tools keeping in mind the following expectations: Reduction in test cycle time and effort, which in turn should result in accelerated time-to-market and reduction in the total cost of testing Reduction of manual intervention Improvement in test case execution repeatability Ability to provide parallelism in testing Simulation of realistic load generation (load ramping) Capability to carry out result analysis and logging for all tests results Achieving the maximum degree of test coverage Support for product evolution, i.e. the testing framework should support future extensions to the test hardware and products.

JUnit Framework with Selenium is the automation tool that has been developed and designed keeping all the above points in perspective. This project aims to briefly review some of the available automation tools and then explain the design and operating characteristics of the framework in detail. The intended users for this project includes both software developers who are engaged in developing similar tools, or prospective users of the framework who wish to analyze how well-suited this framework is for the purpose of their testing.
6

2.2 Gaps identified from the existing systems The existing system is manual or usage of the automated testing tools like win runner and load runner etc. Manual Testing takes more time efforts for a window-based or a web-based application. Even the automated testing tools like win runner have some limitations. All the test cases that are related to the software must be tested thoroughly in order to avoid the upcoming errors during the execution of the software. Most of the test automation frameworks are propriety and can be used with license but the organizational requirements keep on changing. For example QTP is widely used automation framework but it supports only VB. So it better to use open source because it is easy to migrate from one framework to another.

2.3 Proposed solution

The traditional testing is intended to check the application by putting it through a defined series of steps and by analyzing the result after each step. But the manual testing is cumbersome and slow. So the steps are organized so that it can exercise all the functionality of the application as well as reiterate the scenarios that have identified with the problems. The typical testing stages in the product lifecycle are:

Figure 1: Test lifecycle

Selenium has the record and playback functionality which can potentially automate the actions of test engineers. In this the commands also known as selenese in Selenium terminology are written by the test engineers that will automate the application. The various level of information will be provided by the selenium. For this purpose Selenium IDE is used. It has huge collection of selenese which can be used for almost every web component. Moreover we can also write customized command which can be easily integrated with IDE.

Selenium finds its application in automating the black-box/system testing (which encompasses functional testing) of the product under test also known as AUT (Application under test). Selenium can be particularly useful in the regression testing stage. Regression testing involves executing a pre-defined battery of tests against successive builds of the system under test to verify that bugs are being fixed and features / functions that were working in the previous build haven't been broken. Regression testing is an essential part of testing, but is very repetitive and can become tedious when manually executed build after build. The real benefits of using Selenium can be reaped when applied for periodic regression testing.

The UI acceptance tests can be written using JUnit which enables the engineer to run the complete application as a single test from IDE (Eclipse in reference to this paper). So it can be used for acceptance testing also.

Analysis and Design

3.1 Brief Introduction Re-usable test automation frameworks coupled with open-source tools and technologies is a key solution to shrink test cycle times and related costs. This project covers an approach for creating a automation framework around SeleniumRC which is robust, flexible, and extensible. Our framework show increased productivity and ROI, reduced learning curve and dependency on skilled resources, eliminated usage of commercial tools and manual configuration. The framework provides ease of script generation & management, concurrent and remote execution, test data generation and customized reports with screenshots. Preliminary statistics indicate that the automation framework reduces ~ 40% of total test execution effort per release cycle under a given scenario.

3.2 Requirement Analysis 3.2.1 Functional Requirements

Functional requirements capture the intended behavior of the system. This behavior may be expressed as services, tasks or functions the system is required to perform. In product development, it is useful to distinguish between the baseline functionality necessary for any system to compete in that product domain, and features that differentiate the system from competitors products, and from variants in your companys own product line/family. Features may be additional functionality, or differ from the basic functionality along some quality attribute (such as performance or memory utilization). 3.2.1.1 Product Perspective

The product should be easy to use and covers all the functionalities. The product is responsible for regression, unit and acceptance testing of all the updates and service packs for the web client. No changes in code will be required providing build is same. It will help in versioning different build execution on the basis of time and release. We have implemented one click automation technique. This one click can be triggered with or without human intervention. The user just has to parameterize the test cases. The Execution can be triggered periodically.

3.2.1.2 Product Features

The main aim of web client is to provide the search capability in the data that is backed up. For example if the customer has backed up very huge amount of data but he/she wants to restore a particular data only; then web client will provide search and retrieve functionality. This framework will automate each and every feature of web client with its own backup set and its own query.

10

Previously after the release of a service pack or an update; the tester has to manual test the acceptance of it. That is very tedious task and it can also have the chances of human error. This Framework will cut down human efforts considerably, and since only we have to pass the parameter before execution, the results will be more accurate.

Manual testing is very resource intensive. It takes lot of time to validate all the functionality. This framework can tackle this problem by running the product without human intervention. The execution can be periodically invoked so we can do it during free hours of office.

The results are very necessary for validation. This framework will generate and document the results in form of reports. These reports can be configured to automatically mail to respective manager and developer who has issued the service pack or update. These reports will have all kind of time consideration.

There is no need to manually running the execution. But it can be triggered whenever there is any change in the environment that is if there is any change in the code etc. It provides intensive logging for troubleshooting purpose. It provides the log at various levels i.e. debug, warning, info, trace, fatal errors.

3.2.1.3 User Characteristics

The intended users of this framework are testers and developers. The code is written in java. But there is absolutely no need for the user to have the knowledge of java. The users just have to change the parameters which are like changing a text file.

The users should have the knowledge of how web client works and should understand conventions and naming in it. Since, if we are using logs for troubleshooting purposes then the knowledge of web client is must.

The reports generated are very graphic so it can be interpreted by anyone easily.

11

3.2.1.4 Operating Requirements

Operating systems: Windows (all variant including servers), Linux, UNIX. The platform should support JVM (Java Virtual Machine). RAM Required: 4 GB Physical memory. Java Version: Latest JDK (Java Development Kit) Maven (3 onwards).

3.2.1.5 Assumption & Dependencies A number of factors that may affect the requirements specified in the SRS include: The workability of the framework depends upon the latency of web client. If there is unwanted delay in the GUI then there should be some changes to be made in the SRS to accommodate new components that can deal with it. Users are assumed to have a fair estimate of job execution times, so that the decision to accept or reject a job is facilitated. In some module, user will like to execute some step manually; in that case one click interface should have to be altered. We can use framework in the form of a batch file. But in case of platforms other than Windows, it wont work. So we have to execute a python file for doing the same operation so SRS will change in that situation.

3.2.1.6 Domain Requirements

Domain of framework will be java based environment. It can be run in any environment like UNIX, Linux and windows with some small changes. The latency while running on different environment may change from platform to platform. So scripts have been provided with proper time to provide compatibility. The Backup is memory intensive process, so the physical memory requirement for environment is bit high. All the components are java based so it is more advisable to have latest JDK installed on machine. We have used JAR files (Java ARchives) in this project. Most important jars are log4j-1.2.16, selenium-server-standalone-2.16.1 and maven12

core-3.0.4. All the tools and applications that are used in this project are open source. Eclipse (Eclipse Foundation), Selenium RC (Thoughtworks and friends), Maven (Apache community), LOG4j (Apache community). The development tools that are used in this project are Eclipse Helios (Integrated Development Environment), Firebug (to attain all the information about the element under consideration)

3.2.1.7 Users Requirements In an organization this framework can be used by Test Team as well as Development team. For example before issuing any official update, the developer can install the update on a test machine and can run the framework to check the stability of the AUT (Application under Test). The Tester or QA (Quality Analyst) can use it for unit, functional, regression and acceptance testing before launching it in to the market as a deliverable or an enhancement. But in case of integration of new component in the existing system, it is best for a tester to test it rather than developer since a tester has the full knowledge of each and every component.

3.2.2 Non Functional Requirements

A system has properties that emerge from the combination of its parts. These emergent properties will surely be a matter of accident, not design, if the non-functional requirements, or system qualities, are not specified in advance. It includes constraints and qualities. Qualities are properties or characteristics of the system that its stakeholders care about and hence will affect their degree of satisfaction with the system. Constraint is a restriction on the degree of freedom we have in providing a solution.

3.2.2.1 Performance Requirements

It will cut down hours of manual testing into 3.5 hrs. With maximum test coverage. There will be no need to monitor the AUT but it will run by itself. JUnit has evolved with highly defined Exception Handling. Moreover we can isolate each component so it satisfies the basic purpose of Unit testing.
13

We can define our own report format by the configuration of pom.xml in maven. Logging configuration is highly flexible. We can generate highly informative logs with the help of conversion Pattern feature in the configuration file of Log4j. Even the logs can be stored and shown with much variation like it can be stored as a file or it can be stored at a remote server using socket appender etc.

Completeness in testing of all features of the product under test. Easy-to-use interfaces for the user through which the user can enter information into the tool and get both, at-a-glance, as well as, a detailed view of the progress of the test execution.

All the changeable data is stored into easy to read configuration which can be easily modified by the user who doesnt have the knowledge of programming.

It also ensures that every functionality has been tested properly.

3.2.2.2 Safety requirements There are as such no safety requirements. But if we store logs on a network location then it will be necessary that the log size doesnt grow too much. Otherwise it will slow the speed of the machine exponentially.

3.2.2.2 Security requirements There is no such requirement as it is used in organization.

3.3 Design of the proposed system 3.3.1 Design 3.3.1.1 Design Overview The main aim is to design time and space efficient code that can provide full test coverage and high flexibility with respect to inputs. The design includes: 1) Proper parameterization: The automation will work on static dataset so the parameterization should be controlled in such a way that automation will check for all the functionalities.
14

2) Generation of log files: We can generate the logs for future reference with different levels of information like debug, info etc. and can store them on local file system or on network using the socket appender. 3) Generation of the test reports: The framework will generate the reports at the end of each test run. 4) Load Handling: Using this framework we can have simulation of realistic load generation (load ramping).

This framework will realize zero touch automation where there will be no human intervention. This is more flexible and provides high speed automation. The code is written in such a way that it wont have to be modified in case of changes only the external files have to be changed; this is known as data-driven automation.

3.3.1.2 Requirements Traceability Matrix The concept of Traceability Matrix is to be able to trace from top level requirements to implementation, and from top level requirements to test. A traceability matrix is a table that traces a requirement to the tests that are needed to verify that the requirement is fulfilled. A good traceability matrix will provide backward and forward traceability, i.e. a requirement can be traced to a test and a test to a requirements. The matrix links higher level requirements, design specifications, test requirements, and code files. It acts as a map, providing the links necessary for determining where information is located.

15

Figure 2: Requirement Traceability matrix

16

3.3.2 System Architectural Design 3.2.1.1 Chosen System Architecture The system architecture is shown as below.

Figure 3: Automation Framework UML Diagram

17

The system architecture consists of following modules:

1. Structural coding of test cases as suite: Depending on the scenario to be automated the structure of the test suite is defined. There should not any race condition otherwise the results will be misinterpreted. So it is very necessary to structure the test suite. 2. Modification in Selenium RC configuration: As the selenium RC can work with different browsers but we need to change the code to a very little extent thats because different browsers follow different architecture. So this is controlled from the external property file. 3. Selection of delay between selenium commands: It is very necessary to select the delay between Selenese commands because sometime web application takes time to load the page; so selenium executes the another command but element does not exist yet. So the delay between the selenium commands has to be wisely chosen. 4. Identification of the locator: In selenium there are various methods to select the locator; but in some architecture the id of element changes with each execution. So the element will not be located if the previous id is used. So in order to tackle this problem we use Xpath. Xpath is the unique absolute or relative locator with respect to the very first tag. Other methods are css selector and DOM selector. This has to be kept in the mind that one locator can break in another. For example Xpath breaks in Internet Explorer but works fine in other browser like Firefox, Opera, Net Scape, and Safari; so in IE we use css selectors. 5. Reporting: Maven surefire plugin is used to generate the html reports. These reports are generated after the execution gets over. The generation of reports can be controlled or initiated from the command prompt or using batch file. 6. Build System: Maven also provides a build system which control project path and class path. It creates a repository that contains all the jars. The complete build can be controlled using a XML file i.e. pom.xml. We can add any number of plugins for various features based upon our requirement. 7. Logging: The log files are generated using Log4j which is a jar file.

18

3.2.1.2 Discussion of Alternative Designs JUnit vs. TestNG JUnit 4.0 is enabled by the use of annotations. TestNG established is an annotations-based framework long before JUnit. In fact, TestNG pioneered testing with annotations in Java programming, which made it a formidable alternative to JUnit. Mostly both the framework can be used as complimentary to each other. JUnit has always been a unit-testing framework, meaning that it was built to facilitate testing single objects, and it does so quite effectively. TestNG, on the other hand, was built to address testing at higher levels, and consequently, has some features not available in JUnit.

Maven vs. ANT Maven encourages best practices and some people can prefer Ant. Maven is less flexible than Ant. Maven is not just a building tool, it provide an organization and a structure layer. All you do in ant can be done in Maven because you can call ant target.

The main advantage to migrate to Maven is: standardization of how to handle a project, dependencies are clearly defined, the concept of artifacts that can be shared to other projects, documentation and reports, remote and local repository concepts. Maven will help you with: Making the build process easy and provide uniform build environment. Generate quality project information. Use of repository is easy.

ANT will help you with: "Situational Software" which has to be developed quickly (few weeks / months). Projects with external dependencies which are working with "cutting edge" libraries. There is no need for finer grained dependency management. Legacy projects which do not fit the maven conventions very well. Violating maven conventions may become a hassle. Projects without explicit requirements for modularization. The deployable output will consist of only few archives.

19

3.3.3 System Interface Description 3.3.3.1 Detailed Description of Components

A Framework defines a set of guidelines for all phases of test automation: Requirement Analysis, Script Design, Execution, Reporting and maintenance. A framework is a wrapper around some complex internal architecture which makes it easy to use for the end user. It also enforces a set of standards for implementation. Automation framework works on various phases; for each phase there is a component in order to realize it. They are: Phase 1: Script Design: In this phase, assembly of the test scripts is defined. For this purpose our framework uses JUnit. JUnit is highly structural in nature and provide a continuum in the test run. Phase 2: Execution: In this phase, the script is executed. The script is the continuum of various test cases. So in this execution we make sure that if there is an error in any test case, whole script is not broken or stopped. For this purpose our framework uses Selenium RC (Remote Control) which is an open source package of JAVA libraries with jetty as an embedded webserver. Phase 3: Reporting: In this phase, all the information about the automation is generated as a report. This is one of the most phases because it gives the detail information about the success and failure of test run. For this purpose Maven is used. Phase 4: Maintenance: In this phase, information like debug, info, warn, stack trace etc. is generated. This information is used to locate the errors if the test run fails. For this purpose LOG4j is used which also a JAVA library.

20

JUnit JUnit has a graphical user interface (GUI), making it possible to write and test source code quickly and easily. JUnit allows the developer to incrementally build test suites to measure progress and detect unintended side effects. Tests can be run continuously. Results are provided immediately. JUnit shows test progress in a bar that is normally green but turns red when a test fails. An ongoing list of unsuccessful tests appears in a space near the bottom of the display window. Multiple tests can be run concurrently. No subjective human judgments or interpretations of test results are required. The simplicity of JUnit makes it possible for the software developer to easily correct bugs as they are found. It works on the principle of Extreme Programming. Although JUnit was originally written for Java, spinoffs have developed for several other programming languages. The entire family of related testing frameworks is called xUnit. JUnit features include: Assertions for testing expected results Test fixtures for sharing common test data Test suites for easily organizing and running tests Graphical and textual test runners We can generate the logs for future reference with different levels of information like debug, info etc. JUnit is open source framework which is intensively used among the testing community. JUnit works on JAVA platform so new functionality can be added just by using library file known as jar files. The structure of JUnit comprised of test cases and test suites that is depicted in figure 2.

21

Figure 4: JUnit Structure Test cases for JUnit are written as Java classes that extend the JUnit framework. These classes have a number of methods, each of which tests a particular function, or unit, of the code. Every Test case is created with a name, so if a test fails, you can identify which test failed. The default implementation of the test case of do nothing is protected void runTest() { } protected void setUp() { } protected void tearDown() { }

22

Figure 5: Template of test case To gain confidence in the state of a system we need to run many tests. Many test cases are interleaved together to form a test suite.

Figure 6: Template of test suite

Selenium RC (Remote Control)

Selenium RC is the collection of jar files. Selenium supports commands to automate various browser based action and functions using selenium commands or Selenese. There is a selenium server running in the background. It uses Jetty Web server which is developed by Apache community. Jetty provides an

HTTP server, HTTP client, and javax.servelet container. The Selenium Server which launches and kills browsers, interprets and runs the Selenese commands passed from the test program, and acts as an HTTP proxy, intercepting and verifying HTTP messages passed between the browser and the AUT(Application Under Test). Client libraries which provide the interface between programming language like JAVA, python, perl, C# and the Selenium RC Server. So we can write
23

the test scripts in Java or other language and run it from there. It has compatibility with almost all the browsers like Firefox, Chrome, IE, Safari, Netscape etc. but scripts need to be having very minor changes because of the architecture of the browser.

Figure 6: Instance of selenium server

24

Maven Maven is a project development management and comprehension tool. It is based on the concept of a project object model: builds, dependency management, documentation creation, site publication, and distribution publication are all controlled from the declarative file. Maven can be extended by plugins to utilize a number of other development tools for reporting or the build process.
The key components of maven are:

Project object model (POM): The POM is a cornerstone model for Maven 2. Part of this model is already built into the Maven engine (fondly called the reactor), and you provide other parts declaratively through an XML-based metadata file named pom.xml. It is an XML file that contains information about the project and configuration details used by Maven to build the project. It contains default values for most projects. Examples for this are the build directory, which is target; the source directory, which is src/main/java; the test source directory, which is src/main/test.

Dependency management model: Maven is particular about how project dependencies are managed. Dependency management is a gray area that typical build-management tools and systems choose not to be specific about. The Maven dependency management model is built into Maven 2 and can be adapted to most requirements.

There are various maven commands to control different lifecycle of a web based application. The

basic concept of Maven is a project. In Maven terms, any directory that has a project.xml in it is a project. When the sub-directories underneath have their own project.xml, they are projects on their own too. Another concept in Maven is that of a repository. The repository holds the artifacts on which your project depends. There are two kinds of repository: local and remote. Both of them can be declaratively set. Unless specified otherwise, the local repository is created in a special directory called ".maven/repository".

25

The basic structure of Maven is depicted in the figure 8:

Figure 7: Maven Structure The minimum requirement for a POM is the following:

project root modelVersion - should be set to 4.0.0 groupId - the id of the project's group. artifactId - the id of the artifact (project) version - the version of the artifact under the specified group

26

Simplest POM is: <project> <modelVersion>4.0.0</modelVersion> <groupId>com.mycompany.app</groupId> <artifactId>my-app</artifactId> <version>1</version> </project> The maven repository is universally shared repository where hundreds of plugins are stored. So we want to use any plugin we just have to give the name, artifact ID, version and group ID and it will download it by itself in the local repository. Maven controls dependency of the libraries; dependencies is defined on the libraries for example if we use log4j the dependencies are jxl, doxia etc. so maven will take care of it.

LOG4j Log4j is an open source logging utility developed by Apache Community. It allows the developer to control which log statements are output with arbitrary granularity. It is fully configurable at runtime using external configuration files. It eliminates the traditional printf statement usage to log. Log4j has three main components: loggers, appenders and layouts. These three types of components work together to enable developers to log messages according to message type and level, and to control at runtime how these messages are formatted and where they are reported. All these three components are customizable and they are defined into properties file. This file can have either .properties or .xml extension. Loggers can have different levels. The set of possible levels, that is: TRACE DEBUG INFO WARN
27

ERROR FATAL

These levels have the hierarchy FATAL has the highest priority and TRACE the least. The typical configuration file looks like: # Set root logger level to DEBUG and its only appender to A1. log4j.rootLogger=DEBUG, A1

# A1 is set to be a Console Appender. log4j.appender.A1=org.apache.log4j.ConsoleAppender

# A1 uses Pattern Layout. log4j.appender.A1.layout=org.apache.log4j.PatternLayout log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n Log4j is not just for logging into console. You can log to files, emails (SMTP server), a remote server or a database. To control the size of the log files we use rolling appender where we specify a maximum size. If the size of the file is more than that the log file will be rolled back and a file will be created. This is important because the automation framework will work without human intervention so if there is some error; the debug log will keep on increasing drastically that can lead to crash of the complete scripts.

Hudson Hudson is a powerful and widely used open source continuous integration server providing development teams with a reliable way to monitor changes in source control and trigger a variety of builds. Hudson excels at integrating with almost every tool you can think of. Use Apache Maven, Apache Ant or Gradle or anything you can start with a command line script for builds and send messages via email, SMS, IRC and Skype for notifications.

28

Hudson finds its use in continuous integration. With Continuous Integration, the system completes a build, test, deploy, and integration in response to every single commit. If a developer in the web application group checks in code at 2 AM, Hudson kicks off a build, run unit tests, deploys the code to a new server, and performs a set of integration tests. If this build fails or the tests encounter an unexpected condition, everyone is notified of this failure.

With continuous integration, no one needs to drop everything and run a release build, these builds are generated every single day, and in the most mature environments, a fully tested and verified system can be deployed to production at any time. In other words, when you automate build, test, and verify using a tool like Hudson you can continue developing your applications without having to wait (or synchronize) on some manual builds, test, and verify process. Making these processes automatic has another important side-effect; it makes the development process more scaleable. When your teams dont have to stop to actively test and collaborate with one another, it is much easier to add additional developers to a project. Without continuous integration you have to stop and synchronize release schedules. With continuous integration you reduce the risk associated with a particular software development cycle. Features of Hudson, The source code repository that you are using. SVN or CVS support are provided out of the box. Under build triggers choose Poll SCM. Hudson will check your repository every hour. If it has changed, a build will be triggered. Select a build step. If you are running a Java project using either ANT or Maven, then its as simple as adding the details of the appropriate script and target here. If not, choose Execute Shell; you can enter shell commands here to carry out your build and deployment tasks. Add an Email Notification post-build action to enable Hudson to notify the developers who were responsible for breaking the build! If youre running a Java app, you can also include all your JUnit test results in the application reports.

29

The following diagram gives the overview of Hudson.

Figure 8: Hudson

Implementation

3.1 Tools Used JUnit JUnit has a graphical user interface (GUI), making it possible to write and test source code quickly and easily. JUnit allows the developer to incrementally build test suites to measure progress and detect unintended side effects. Tests can be run continuously. Results are provided immediately. JUnit shows test progress in a bar that is normally green but turns red when a test fails. An ongoing list of unsuccessful tests appears in a space near the bottom of the display window. Multiple tests can be run concurrently. No subjective human judgments or interpretations of test results are required. The simplicity of JUnit makes it possible for the software developer to easily correct bugs as they are found. It works on the principle of Extreme Programming.
30

Although JUnit was originally written for Java, spinoffs have developed for several other programming languages. The entire family of related testing frameworks is called xUnit. JUnit features include: Assertions for testing expected results Test fixtures for sharing common test data Test suites for easily organizing and running tests Graphical and textual test runners We can generate the logs for future reference with different levels of information like debug, info etc. JUnit is open source framework which is intensively used among the testing community. JUnit works on JAVA platform so new functionality can be added just by using library file known as jar files.

Selenium RC (Remote Control)

Selenium RC is the collection of jar files. Selenium supports commands to automate various browser based action and functions using selenium commands or Selenese. There is a selenium server running in the background. It uses Jetty Web server which is developed by Apache community. Jetty provides an

HTTP server, HTTP client, and javax.servelet container. The Selenium Server which launches and kills browsers, interprets and runs the Selenese commands passed from the test program, and acts as an HTTP proxy, intercepting and verifying HTTP messages passed between the browser and the AUT(Application Under Test). Client libraries which provide the interface between programming language like JAVA, python, perl, C# and the Selenium RC Server. So we can write
the test scripts in Java or other language and run it from there. It has compatibility with almost all the browsers like Firefox, Chrome, IE, Safari, Netscape etc. but scripts need to be having very minor changes because of the architecture of the browser.

Maven Maven is a project development management and comprehension tool. Based on the concept of a project object model: builds, dependency management, documentation creation, site publication, and distribution publication are all controlled from the declarative file. Maven can be extended
31

by plugins to utilize a number of other development tools for reporting or the build process.
The key components of maven are:

Project object model (POM): The POM is a cornerstone model for Maven 2. Part of this model is already built into the Maven engine (fondly called the reactor), and you provide other parts declaratively through an XML-based metadata file named pom.xml. It is an XML file that contains information about the project and configuration details used by Maven to build the project. It contains default values for most projects. Examples for this are the build directory, which is target; the source directory, which is src/main/java; the test source directory, which is src/main/test.

Dependency management model: Maven is particular about how project dependencies are managed. Dependency management is a gray area that typical build-management tools and systems choose not to be specific about. The Maven dependency management model is built into Maven 2 and can be adapted to most requirements.
There are various maven commands to control different lifecycle of a web based application. The basic

concept of Maven is a project. In Maven terms, any directory that has a project.xml in it is a project. When the sub-directories underneath have their own project.xml, they are projects on their own too. Another concept in Maven is that of a repository. The repository holds the artifacts on which your project depends. There are two kinds of repository: local and remote. Both of them can be declaratively set. Unless specified otherwise, the local repository is created in a special directory called ".maven/repository".
Log4J

Log4j is an open source logging utility developed by Apache Community. It allows the developer to control which log statements are output with arbitrary granularity. It is fully configurable at runtime using external configuration files. It eliminates the traditional printf statement usage to log. Log4j has three main components: loggers, appenders and layouts. These three types of components work together to enable developers to log messages according to message type and level, and to control at runtime how these messages are formatted and where they are reported.

32

All these three components are customizable and they are defined into properties file. This file can have either .properties or .xml extension. Loggers can have different levels. The set of possible levels, that is:

TRACE DEBUG INFO WARN ERROR FATAL These levels have the hierarchy FATAL has the highest priority and TRACE the least. The typical configuration file looks like: # Set root logger level to DEBUG and its only appender to A1. log4j.rootLogger=DEBUG, A1

# A1 is set to be a Console Appender. log4j.appender.A1=org.apache.log4j.ConsoleAppender

# A1 uses Pattern Layout. log4j.appender.A1.layout=org.apache.log4j.PatternLayout log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n Log4j is not just for logging into console. You can log to files, emails (SMTP server), a remote server or a database. To control the size of the log files we use rolling appender where we specify a maximum size. If the size of the file is more than that the log file will be rolled back and a file will be created. This is important because the automation framework will work without human intervention so if there is some error; the debug log will keep on increasing drastically that can lead to crash of the complete scripts.

33

Hudson Hudson monitors executions of repeated jobs, such as building a software project or jobs run by cron. Among those things, current Hudson focuses on the following two jobs: 1. Building/testing software projects continuously, just like CruiseControl or DamageControl. In a nutshell, Hudson provides an easy-to-use so-called continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity. 2. Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Hudson keeps those outputs and makes it easy for you to notice when something is wrong. 4.2 Implementation We have used agile development. Agile software development uses iterative development as a basis but advocate a lighter and more people-centric viewpoint than traditional approaches. Agile processes use feedback, rather than planning, as their primary control mechanism. The feedback is driven by regular tests and releases of the evolving software. This automation framework works on Extreme Programming (XP), the phases are carried out in extremely small (or "continuous") steps compared to the older, "batch" processes. The (intentionally incomplete) first pass through the steps might take a day or a week, rather than the months or years of each complete step in the Waterfall model. First, one writes automated tests, to provide concrete goals for development. Next is coding (by a pair of programmers), which is complete when all the tests pass, and the programmers can't think of any more tests that are needed. Design and architecture emerge out of refactoring (disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior), and come after coding. The same people who do the coding do design. (Only the last feature merging design and code is common to all the other agile processes.) The incomplete but functional system is deployed or demonstrated for (some subset of) the users (at

34

least one of which is on the development team). At this point, the practitioners start again on writing tests for the next most important part of the system.

Figure 9: Agile Development Process Model

35

Figure 10: View Process Model

36

4.2.1. User Interface Design The Automaton framework can be executed using a batch file or it can be run from Hudson. The sequence diagram is:

Figure 11: Sequence Diagram

37

4.2.1.1 Description of the User Interface There are mainly two phases of the Automation Framework: Phase 1: In this phase, framework will interact with the actual data. It will check for job id and the data associated with it. If there is a job already ran. It will delete that job and prune all the data that is being associated with it. The main operation that is taking place in this phase is to back up data and populate that data in the Web client. To maintain the integrity of data, the environment has to be cleaned. For this phase python scripts are being executed with java. Phase 2: In this phase selenium command will run and all functionality will be automated.For both the phases, logs are maintained. But the reports are generated only for phase 2.

4.2.1.2 Screen Images

Phase 1:

Figure: Job Controller

38

Figure: Configuration options

Figure: Storage Policy

39

Phase 2:

Figure: Hudson configuration

40

Figure: Project in Hudson

41

Figure: Run options in Hudson

Figure: Login to web console

42

Figure: Logs

Figure: Search page

43

Figure: Advance Search Options

Figure: Query Builder

44

Figure: Search Options

45

Figure: Details of Search

Figure: Job Status

Figure: Summary of Packages

46

Figure: Summary of Test Suite

47

Figure: Failure Details

Figure: Details of Plugins

48

Figure: Logs

4.3 Testing 4.3.1 Test approach Unit testing is usually conducted as part of a combined code and unit test phase of the software lifecycle, although it is not uncommon for coding and unit testing to be conducted as two distinct phases. Field testing will be performed manually and functional tests will be written in detail. Test objectives are: All field entries must work properly. Pages must be activated from the identified link. The entry screen, messages and responses must not be delayed.

49

Integration Testing Software integration testing is the incremental integration testing of two or more integrated software components on a single platform to produce failures caused by interface defects. The task of the integration test is to check that components or software applications, e.g. components in a software system or one step up software applications at the company level interact without error. 4.3.2 Test Plan 4.3.2.1 Features to be tested

The time and space complexity of framework is to be tested. That is done using test coverage and maven reports. The code is tested against a battery of expected and non-expected delay. This testing is done to check for the integrity of framework against changing environment. The framework is tested for many different inputs and the output is tested against the output obtained from manual testing. The integrity of code is tested using Eclipse debugger. The framework is tested against various components it is using like maven, log4j etc.

4.3.2.2 Features not to be tested

The report generation is not to be tested because it is taken care by maven- surefire plugin. But the configuration is tested effectively. The integration of complete framework as an environment is not to be tested.

4.3.3 Testing tools and environment

Tools that are used for testing purpose is Windbg (Debugger provided by Microsoft), Eclipse IDE (when the framework is run into debug mode). For the maintenance purpose all the debug processes are logged into logs-Debug.log. This log gives us the information about everything that is running in background.
50

Microsoft Depends is used to check for dll files.

4.3.4 Test Cases Cases Purpose Inputs Expected Output 1 To check if the backup instance is started properly using python. 2 To check if selenium is able to identify the elements. 3 If embedded jetty is able to send the http request and get the response using selenium server. 4 To check if all the elements of web client has continuity of actions. 5 To check if reports are generated properly with adequate
51

Status

Parameters for backup.

Exit code=0

Passed

Xpath as locator. Elements are located properly.

Passed

Port number.

Browser has started successfully.

Passed

Inputs for dropdown, input boxes etc.

The actions are in sequence.

Passed

Test suite, test case.

The reports are generated successfully.

Passed

information. 6 To check if the logs are getting populated successfully. 7 To check if the environment is clean. 8 To check if http requests are stopped properly and port is free. Table 1: Test Cases Port number. The browser has stopped successfully. Passed Input from framework. The environment Passed is clean. Output from framework is used as input. The logs are successfully written. Passed

4.3.5 Test Procedure All the test cases given above are tested in the same sequence. If there is any error in the predecessor test case then execution will stop and test case will be considered as failed and all the information will be provided in the logs.

5 Results and Discussion

5.1 Results

The automation framework has been proposed and implemented successfully. The framework will be very helpful in tedious manual testing of any service pack and update that is being issued. The framework will find its use in functional, regression, unit and integration testing of web client with no requirement to monitor it. All the information will be stored in the form of highly qualified reports. So there will be no requirement for a tester to document each run.

52

5.2 Performance Analysis The framework in preliminary usage is saving almost half of the time of manual testing. All the components are working properly. There will be no requirement to document testing of web client. The reports generated by framework will be sufficient. It will huge amount of time because it is not necessary to monitor the run. So we can use framework during no-working hours. The time that is saved by the framework can be utilized in other important tasks.

The performance is analyzed as: Automatic Test Execution: The prototype implemented had only all features that were absolutely needed, analyzing the outcome and reporting results could be automated fairly easily. Ease of Use: Usability is always a subjective matter but creating test cases with the framework proved to be easy both for me and also for those people I introduced the prototype. Even creating user keywords was considered fairly straightforward. Maintainability: The prototype framework itself was surprisingly small which suggests that it ought to be pretty easily maintainable. For anyone using the framework that is not the important issue since they see it mainly as a black box. The much more important thing is that the test libraries and driver scripts required for each run were both very small and relatively easy to read which makes them easy to maintain. Based on the runs it also seemed that separating the test data from the automation code really eased maintainability of both the code and test cases constructed in the data.

5 Conclusion and future enhancements

Automation is an important aspect of todays software industry. It saves a lot of time and man power. In the competitive market where the needs are very dynamic and time to fulfill the requirement is less, the automation framework comes very handy and efficient. Automation is an evolving process. If there is more effective tool in the market that can be used for automation then migration will be one of the options. For current framework, we can provide

53

plenty of other information like memory usage, test execution pattern information, more efficient reports etc.

8. References
[1] The official website of Log4j for jar downloads and documentation http://logging.apache.org/log4j/ . [2] Software Engineering by IAN Somerville for documentation. [3] The official site for maven for information of http://maven.apache.org/. [4] For selenium documentation, commands, downloads http://seleniumhq.org/ [5] Apache Maven 2 -Effective Implementation, Build and manage applications with Maven, Continuum, and Archiva Written by Maria Odea Ching, Brett Porter [6] Pro Apache Log4j, Second Edition by Samudra Gupta [7] Selenium Testing Tools by David Burns [8] Test Driven Practical TDD and Acceptance TDD For Java Developers By Lasse Koskela

54