Business Value of Agile Project Management

Final Paper Submission to QAI 2011

By

Krishna Murthy P
Deloitte Consulting India Pvt. Ltd. HYDERABAD

Email @ kpradhan@deloitte.com
Abstract –
Enterprise level product development is growing beyond the boundaries of silo-ed enterprise applications to meet the
requirements of rapidly changing markets. They are getting integrated within and across the enterprise thereby
metamorphosing into ecosystems. This has resulted in complexity which seems to be ever increasing and also in a high number
of changes these systems have to go through. Nevertheless to say, the tester’s job gets more complex in propionate to this as
well. For example, when the product is claiming to scale 1000 or more nodes/users/transaction/browser instances, the
organization should be in the position to test and certify accordingly. Most of the time product/applications fail due to lack of
the infrastructure.
The above mentioned challenges can be tackled by shifting the organizational IT landscape to Cloud computing. Focus should
be on Cloud based testing with niche capabilities in the areas of performance testing, security testing, reliability testing, and
experience in virtualization technologies. Cloud Computing is an innovation we can apply to bring economy to our solution in
terms of time, resources and of course, the money!
Applying Cloud based testing facilities can help us accrue many benefits such as Reduction in Cost – Capital Expenditure
(owing to zero infrastructures), Reduction in execution time => Rapid Feedback cycle, Quick Turn-around => Increase in
Productivity, Reduction in Risk => Increase in Quality

Introduction–
The recent sharp downturn in the economy is forcing organizations to reconsider their approach towards IT investments. In a
world, where companies are more focused towards improving efficiencies and return of capital employed, one has need to re-
consider how they can reduce their technology investments, or get higher return on the same or incremental investments.
Testing is crucial to enhance user satisfaction and reduce support cost. However, testing requires organizations to invest in
people, tools and environments and can take up a significant percentage of the available budget. But quality can never be
compromised. New ways of development and testing are enabling organizations to ensure higher quality but with significantly
lower investments. These challenges can be tackled by shifting the organizational IT landscape to Cloud computing. Focus
should be on Cloud based testing with niche capabilities in the areas of performance testing, security testing, reliability
testing, and experience in virtualization technologies. Cloud Computing is an innovation we can apply to bring economy to our
solution in terms of time, resources and of course, the money!

Objectives –
Here are the key objectives; we are going to focus while we detail the rest of the document.
 To bring a better quality product on Cloud (Ensure the Product is ready and can work on Cloud Environment)
 Ensure the required planning is done in terms of testing cycle for the Product before moving to Cloud
 Ensure the business benefits of a Product Testing on Cloud and its related activities accomplishments
 Help educate and prepare the Testing Team with required knowledge and skills to prepare for Cloud Testing

Key Challenges for Product testing on Cloud –

Data Governance
In essence, enterprises have a ton of sensitive data that requires access monitoring and protection. Data (and information
generated from the data) is the life blood of many enterprises; the loss of control will not be acceptable. On top of all that,
enterprise must comply with many of the regulations that require data governance. By moving the data into the cloud,
enterprise, for now, will lose some capabilities to govern their own data set. One has to rely on the service providers to
guarantee the safety of their data. So who’s tackling this problem? It is really up to the service providers such as Amazon,
Google and Sales force to provide guarantees that customer data are safe and access to data are restricted and protected. As a
tester, one has to ensure that the Product and interfacing system are data protected while testing on Cloud.

Manageability
Most of these Cloud computing vendors, lack required infrastructures and platforms that do not have great management
capabilities. This is not unusual. Throughout computing history, raw capabilities will generally appear on the market first, and
then management of these raw capabilities becomes a differentiator when competition heats up. An example of missing
management capabilities for cloud infrastructures is auto-scaling. Amazon EC2 claims to be elastic; however, it really means
that it has the potential to be elastic. Amazon EC2 will not automatically scale your application as your server becomes heavily
loaded. It is still up to the developer to manage that scalability problem. So who’s tackling this problem? As a tester, one has to
keep management team in loop build required infrastructure on the Product to be self scaling and Cluster enabled.

Monitoring
Monitoring, whether is for performance or availability, is critical to any Enterprise. We are not talking about just how much
CPU or memory the machines are using. We are talking about performance of transactions and disk IO and others. CPU and
memory usage are misleading most of the time in virtual environments. The only real measurement is how long your
transactions are taking and how much latency there are.

According to High Availability’s article on latency:

• Amazon found every 100ms of latency cost them 1% in sales.
• Google found an extra .5 seconds in search page generation time dropped traffic by 20%. A broker could lose $4
million in revenues per millisecond if their electronic trading platform is 5 milliseconds behind the competition.
So who’s tackling this problem? We need to develop APIs around the Product with capabilities for monitoring Amazon’s web
services, or monitoring for Google App Engine.

Scalability of the product:

When we are claiming that the product can be scale up to n-numbers of node/users/perform transaction, we need to make sure
that it should be thoroughly tested before it goes to the end users. It will be challenge to the tester to develop the program or
look for tool which can help to generate the scalability matrix based on the numbers of nodes/users/transaction/browser
instances.

Reliability and Availability

I won’t beat the dead “Gmail down, EC2 down, etc down” horse here. But the truth of the matter is enterprises today cannot
reasonably rely on the cloud infrastructures/platforms to run their business. There’s almost no SLAs provided by the cloud
providers today. Can one imagine enterprises signing up cloud computing contracts without SLAs clearly defined? It’s like
going to host their business critical infrastructure in a data center that doesn’t have clearly defined SLA. We all know that
SLAs really doesn’t buy you much. In most cases, enterprises get refunded for the amount of time that the network was down.
No SLA will cover business loss. As long as there’s a defined SLA on paper, when the network/site goes down, they can go
after somebody. So who’s tackling this problem? Well, again, no one is today as far as I know. Maybe some of us as a tester
team or community have to ask for this before agreeing to deploy the product on Cloud or before we start testing it.

Cloud Testing – Infrastructure Matters

A firm wanted to test their application and need 100 browser instances. In the old days it would have required 100 machines -- that would be
a massive undertaking. Even with hardware virtualization, you would need 5 to 10 machines, and there would be some complex
configuration issues. However, by putting it all in the cloud, they were able to sync up 100 virtual instances of the browsers and take them
down over a weekend at a dramatic cost reduction. The truth is that when you start spinning up multiple instances of any client for the
purposes of testing an application you can inadvertently doom your test to fail due to underlying infrastructure attributes that aren’t always
all that obvious to the uninitiated regardless of whether those clients are in the cloud or not.
Basically the most important thing to remember is that if you’re using a cloud-based testing service or a cloud-based testing environment that
you’re probably testing a production environment and its underlying infrastructure.
Let me reiterate, we are not just testing the application, you are testing its infrastructure, too. And remember that when that traffic is
generated externally that you’re also poking at the infrastructure of the Internet, and may well trigger policies “out there” that you’re ISP – or
others – have set in place to protect your applications and network.
It’s therefore important to understand what policies may be triggered by sudden bursts of traffic as well as what security policies may be
triggered by the constant hammering of a single client at an application. It’s also important to understand the flow of requests and responses
through the infrastructure and recognize the various touch points along that path. Doing so may ease the troubleshooting process greatly.
Testing applications in a lab or even a QA environment has its own pitfalls as you are testing only the application and as we all know, no
application is an island. But testing them in a production environment, while ideal for gathering a real view of how the application will
perform when faced with users and customers, has its own challenges because you’re testing against a real, live environment with all the
mechanisms in place to ensure quality, performance, and security of those applications.
We need to gather up the right folks before you start testing and understand the potential pitfalls and how to properly address them – or avoid
them – before you spend money firing up hundreds of instances of clients in the cloud or you may find yourself wasting more money than
you might have just testing them on your own.

Approach for Cloud based Testing

Before you invest resources in the cloud, it's important to set your priorities and determine which features are most important
to your enterprise. Weigh each of the following according to your enterprise’s needs.

Autonomy determines how the service functions on its own, including any dependencies that may be present.

Integration testing is required to see how the service works when leveraged by other systems, systems that perhaps are
known or unknown at the time of development.

Granularity testing determines if the service was created with too much of a coarse- or fine-grain leaning, which has
an effect on performance and the value of a service.

Stability testing insures that the services built won’t fall down at the worst of times. This is usually simple regression
testing, with some integration testing here, as well.

Performance testing is just what you would expect: the ability to determine if the services can handle many
simultaneous requests, and any special architecture that may be required to insure good performance, such as load
balancing with transactions.

POC approach

In Sever-Agent-DB enterprise based architecture, to start with we need to do the POC for 3-tier applications (with few servers say 3 servers,
4 agents) and few scaling brokers with environment on Cloud. If it gets deploy without any blockage then we can go for deploying the large
no. of agents or connection with n-users.
We need to make sure that data is secure and encrypted when the transition/updating is happening from the Database and Server level
components. If the communication is not secure but the data is encrypted then it is safer to communicate from the cloud configure machines
to local premises machines
Secure Data communication and synchronization

Keeping the Database on the cloud configure environment once the transaction is over then the database can be replicate to the local
premises so that the server is getting the information from the database should be very fast and the it is secure.
Creation of test lab/test scenarios

We need to decide which part of testing is going to be in cloud, based on that we need to prepare the specific test scenarios to
validate functionality of the features in the application. For better approach to make small prototypes based on the components and
in the same time prepare cloud based test lab to test the developed prototype. The real scenarios of testing an application it based on
i.e. high end machines, Database can be configured on Cloud and other machines like Servers and Agents (say in 3-tier applications)
can be deploy on the local premises.

Product/Application Testing on Cloud – Detailed approach (PoC)

This proof of concepts I have conducted on real product while I was leading the Testing Team in CA, Hyderabad during Jan – Feb 2010 time frame.
Scope:

The scope of this document is to prove that ACM (Application Configuration Manager) product can be compatible with Cloud environment by testing the
various scenarios on Cloud. To test the various scenarios we have considered 6 ACM agents, 2 Grid nodes and 10 components keeping in/out of the Cloud.
Based on the scenarios estimation is made using Amazon Machine Images from Amazon EC2.
As the current ACM product requirement is to scale more than 10000 servers, and it is very hard to bring up the lab for testing the scaling feature. We need
to think of the cost before moving and testing the product on Cloud. This document explains the estimation based on proof of concept of ACM product on
Cloud.

Configuration:

ACM Server: One physical machine with 4GB RAM 80GB disk space and Core 2 duo processor with Windows 2003 Server

Amazon Machine Images (AMIs): Amazon EC2 Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit),
160 GB of local instance storage, 32-bit platform, I/O Performance: Moderate

ACM Agents: Six physical machines with 2GB RAM 80GB disk space and Core 2 duo processor with Windows 2003 Server and Red hat Linux

Amazon Machine Images (AMIs): Amazon EC2 Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit),
160 GB of local instance storage, 32-bit platform, I/O Performance: Moderate

Grid nodes: Two physical machines with 2 GB RAM 40GB disk space and Core 2 duo processor with Windows 2003 Server

Amazon Machine Images (AMIs): Amazon EC2 Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit),
160 GB of local instance storage, 32-bit platform, I/O Performance: Moderate

ACM Database: One physical machine with 4 GB RAM, 160 GB disk space and Core 2 duo processor with Windows 2003 server and SQL Server 2005

Amazon Machine Images (AMIs): Amazon EC2 Large Instance (Default) 7.5 GB of memory, 2 EC2 Compute Unit (2 virtual core with 2 EC2 Compute Unit
each), 420 GB of local instance storage, 32-bit platform, I/O Performance: High

Managed Components:

1. Windows 2003 Server

2. Red Hat Linux Server
3. Microsoft SQL Servers 2005
4. Oracle Database 11g
5. Oracle Weblogic
6. IBM Websphere
7. JBOSS
8. Java Application Server
9. IIS Server
10. Apache HTTP Server
Operations that are planning to execute on cloud:

1. Discovery operations
2. Refresh operations
3. Change detection operations
4. Compare Server operations
Test Scenarios:

1. Deploying Grid nodes in the Cloud

ACM Server, ACM Agents and Databases in the local premises

ACM Server, Agents, DB operated Locally and Grid nodes in the cloud

DB

ACM Server

Grid Nodes

2. Deploying Grid nodes and Database in the Cloud

ACM Agents and ACM Server local premises

ACM Agents
 ACM Server and Agents operat
3. Deploying ACM Remote Agents, Grid nodes and Database in Cloud

ACM Server is located in local premises

ACM Server operated Locally and ACM Agent, Grid nodes and DB in the cloud

DB Grid Nodes

ACM Server

ACM Server
4. Deploying Grid nodes, ACM Remote Agents in Cloud

ACM Server and Database in local premises

ACM Agents

ACM Server and DB operated locally and ACM Agents and Grid nodes placed in the cloud

ACM Server Grid Nodes

5. Deploying Grid nodes, ACM Server, DB in the Cloud

ACM Agents is in the local premises

ACM Agents

DB

ACM Agents operated Locally and ACM Server, Grid nodes and DB in the cloud
ACM Agents

Grid Nodes DB

ACM Agents
Estimation:

Based on the current setup (i.e. without Cloud) I have estimated the time it takes to perform below mentioned operations:

Operations Time
estimation

Discovery 00:55 Mins

Refresh 00:26 Mins

Snapshot 00:06 Mins

Change Detection 01:15 Mins

Compare Server 01.25 Mins

Estimation to support the ACM product on Cloud (Amazon EC2)

Estimation is taken using this link http://aws.amazon.com/ec2/instance-types/ based the instances are going to use.

As of now Internet data transferred “in” and “out” of Amazon EC2 is free till Jun 30, 2010

10 AMI’s machines are required, out of which 9 machines are Standard small (Default) Instances and one Large On demand Instances. We are assuming
that continuous 4 days are required to test the above scenarios.

Standard On-Demand Instances Linux/UNIX Usage Windows Usage

Small (Default) $0.095 per hour $0.13 per hour

Large $0.38 per hour $0.52 per hour

The cost is associated with the Small default instances for Windows usage: (($0.13 *24)*4)) = $12.48 for single machine.

For 8 Standard on-Demand small instances (Windows Usage), the cost is: ($12.48 * 8) = $112.32

For 1 Standard on-Demand small instances (Linux Usage), the cost is (($0.095 *24)*4)) = $9.12

For 1 Standard on-Demand Large instances (Windows Usage), the cost is: (($0.52*24)*4) = $49.92

So total cost is: ($99.84+$9.12+$49.92) = $171.36

** The above calculation is based on 4 days continues 24 hrs per day.

Dependencies:

ACM product is integrated with other CA products like:

• BO (Business object) reports

• NDG (Network discovery gateway)

• CMDB

• EEM (Embedded Entitlements Manager)

So while testing the ACM product on Cloud we need to make sure that other product also supports Cloud environment.

Risks Encountered:

 How secure is ACM product when exposing to Cloud.

 Dependences on other products
 Cost associated while testing the scenarios in case of delay due to unforeseen problems which may come during runtime.
This POC helped us to conclude that the “ACM” product is Cloud compatible.

Guidelines for Testing a Product/Application on Cloud – Do’s and Don’t

Constantly Test Everything – the Right Way

The Cloud demands that we be as nimble as possible, delivering features and fixes in almost real-time fashion. Both customer and provider rely on
software development that can maintain quality while being light on its feet and constantly moving. In addition, Cloud-oriented systems tend to be highly
complex and dynamic in structure -- more than our industry has ever seen before. The traditional software development and testing models do not support
this constant “diligence in motion”; therefore a new Cloud delivery model must be adopted. Traditional models worked reasonably well in the world of
client / server, since users were most often internal resources. User experience was downplayed, and glitches tolerated.

The lengthy cycle for requirements generation and feature development, followed by a set of testing cycles, allows for extended periods of time without
testing. But these gaps do not correlate with the needs of Cloud consumers. For them, ongoing, reliable, uninterrupted experience is everything.

The only way companies can realistically achieve this model is to have superior test sets, that are fully automated – and to go about automation the right
way. Otherwise it can quickly become unachievable and unmanageable. When automation efforts fail to achieve high percentages on tests, the method is
often considered faulty. But when test automation follows specific and unique guidelines, its success can be measured again and again.

Guidelines for Successful Cloud Test Automation

When an automation team spends a disproportionate amount of time on automating tests, and the resulting automation then ends up covering only about
30% of the tests, the automation policy has failed. A much higher percentage is needed to "test everything always” in Cloud applications. Additionally,
automation should not dominate the test process. The actual automation development, and more importantly maintenance effort should only have a modest
footprint in terms of time and resources. While many testing organizations mistakenly approach automation from the perspective of tooling or
programming, an approach centered on automation effective test design combined with an agile test development process yields far better results. When
done right, the result is a set of automated tests with on-the-fly adaptability that readily meets the facile requirements of testing in the Cloud.

Tools have their place in the process, but frequently steal the center of attention, viewed as panaceas. Primary focus goes to buying and learning “the tool”
rather than expending the time, effort and cost involved in revisiting test design. Establishing a test design process allows for more possible tests that are
readily available, improving development cycles through flexibility. The approach aims to have at least 95 percent of tests automated, and 95 percent of
testers’ time spent on test development, not automation. These tests are not based on regression or bug validation, but are calibrated to find and hunt for
bugs, boundary conditions, state transitions, exploratory testing and negative tests.

Define your Paradigm

Today's companies increasingly find that they are in an ever more competitive market, especially in the drive to implement more robust, capable and
pioneering Cloud-based products and services. Product delivery times are decreasing, customers demand higher and higher levels of product quality, and
failure to deliver within the customer's expectations can be swiftly punished with whole scale product abandonment and erection of barriers to market
reentry.
Adopting a testing paradigm that is designed specifically for the requirements of Cloud-computing is a fundamental requirement for the new standards of
quality being set by customer-driven demand.

1. Evaluate Your Current Testing Paradigm

Each quality or development team has its own frame of reference by which it identifies the need for testing overall, and the specific kinds of
testing required. This frame of reference, or paradigm, often reflects the higher organizational approach to quality assurance and the role that
testing is perceived to occupy in customer satisfaction and loyalty.
Cloud computing introduces a need for a new paradigm - one that incorporates the 24/7 testing-on-the-fly required for successful Software as a
Service implementation and other Cloud-based products.
Some organizations may find their current approach to testing adapts relatively well for the "test everything all the time" model. However, most
will find it necessary to reassess not just their approach toward testing, but the fundamental principles that underlie that approach - ones they've
inherited from traditional testing models.
In the world of software testing, all the players used to start in pretty much the same place: manual testing. As the industry matured, more elegant
testing solutions - record and playback, automated or scripted testing and action-based testing - emerged over time. Each new approach to testing
successively increased our testing efficiencies, reporting efficiencies and ability to handle larger and larger test loads, lending leverage to the
leading few. Conducting a Current-State Assessment can give you the necessary insight into what works and doesn't work in your current testing
paradigm when it comes to testing in the Cloud.
Conducting a Current-State Assessment
Most companies have tried a variety of testing approaches, some manual, some scripted, and maybe even some attempts at automation. It is
important for a firm to conduct an honest self-assessment about their current testing footprint, capabilities, limitations, and opportunities for
improvement.
Many times firm have a belief about their current testing approach and are surprised to find that those beliefs, albeit hopeful, are not fully rooted in
practice. The following questions will help thoroughly evaluate your testing paradigm.
 Testing methodology - Is it largely manual or some combination of methods? Are those methods consistent?

 Testing environment - Where are the applications located? Where is the data located? What is the network and performance constructs?
 People resources and skill sets - What is the competency makeup of your current test team? What are the skill sets required for moving
forward?

 Time-to-deliver constraints - What are the barriers to testing efficiency? What are the efforts that take more time than others and why?
 Reporting and progress visibility - Do you have consistent visibility into your testing status? Are you surprised when testing efforts are
late? Are statuses verbal, or based on actual metrics?

2. Define Paradigm Requirements that Meet Standards for Cloud-based Technologies

Achieving your organization's testing goals, in whole or in part, requires planning and mapping out your "testing in the Cloud" strategy and its
requirements.
Clarifying Cloud Architecture
An initial understanding of the different types of clouds and what their high-level architectures offer will inform the kind of testing paradigm you
will need to establish.
Sam Johnston's integrated picture depicts the three cloud types:

Remote, Public or External Cloud - Public clouds are sometimes referred to as "Regular" cloud computing. Completely separate from a user's
desktop or the corporate network that they belong to, public clouds offer a pay-per-use service model because the user is leveraging outside
compute resources for the particular service they are seeking. This approach offers economies of scale, but their shared infrastructure model can
raise concerns about configuration restrictions, adequate security, and service-levels (available uptime). These concerns might make you think
twice about subjecting sensitive data that is subject to compliance or safe harbor regulations.

Because "public" clouds are typically made available via the public internet they may be free or inexpensive to use. A well known example of a
public cloud is Amazon EC2 which is available for use by the general public.
Internal or Private Cloud - Private cloud computing extends the same infrastructure concepts firms already have in their data centers. The
motivation for private clouds appears to be to resolve security and availability concerns inherent in the public cloud paradigm. As such, private
clouds seemingly are not burdened by network bandwidth, availability issues or potential security risks that may be associated with public clouds.
However, this thinking belies the very intent of cloud computing which is predicated on hardware-software extensibility, dramatic reduction in
infrastructure costs, and an elimination of the management concerns governing private networks.

Mixed or Hybrid Cloud - Many of the leading engineering thinkers in the industry suggest that the most workable cloud computing approach is the
"hybrid" approach. The hybrid solution combines the best of the Public and Private Cloud paradigms. Considering that some applications may be
too complex or too sensitive to risk operating from a public cloud it makes sense for a firm to protect those application and data assets within the
construct of a private cloud where they have total control.

Less sensitive applications and data can be migrated to a public cloud freeing compute resources that can be repurposed for the complex
applications that need to stay home. The hybrid approach does sound like the best of both worlds. It makes sense from a technology and economic
standpoint. It allows for control, flexibility and growth. The trick to managing hybrid clouds comes when you consider spikes in demand.
 When demand spikes pummel the performance of your applications located within the private cloud -and you need additional computing power
(such as is experienced by web-based news media when critical events occur) -you will need to develop a management policy that can be
responsible for when to reach out to the public cloud for those additional resources.
Defining the Future-State
Envisioning the results of your testing transformation requires solid understanding of your organization's business goals and objectives, the Cloud
computing paradigms that may help your testing effort contribute to those goals, and development of a sound plan to move in the new direction.
When documenting your planned future state, address each of these categories:
 Architectural - Consider the different Cloud paradigms as they pertain to your business model, goals and objectives, and application and
data sensitivity.

 Organization - Ensure organizational review and consensus of new Cloud testing direction as it pertains to business goals and objectives
and priorities.

 Financial - Define the benefits, know where the real costs lie, and define the budget.

 Implementation - Adopt an incremental improvement approach, and choose the correct tools and partners.
 Monitor and measure - Develop a consistent set of metrics for measuring and monitoring your new foray into testing in the Cloud.
Other organizational goals and objectives that are important to include:
○ Provide sufficient support for distributed test teams

○ Increase your return-on-testing-investment

○ Significantly decrease time-to-market

○ Optimize the reusability of tests and test automation

○ Improve test output and coverage

○ Enhance the motivation of your testing staff

○ Increase managerial control over quality and testing

○ Have better visibility into quality and testing effectiveness

3. Apply Proven Methods and Technology

Implementing the changes necessary to adopt a new testing paradigm will bear out how well you've defined your requirements. Therefore,
selecting well-matched testing tools and specific architectural approaches can have a dramatic impact on the results of your paradigm shift.
Cloud Accessible Application
This category of cloud computing technology focuses on applications that are entirely externally hosted at a cloud services application vendor. No
installation of software is required on the user desktop. The application functionality is utilized through a web browser and the data is stored on
the application host's servers. Salesforce.com is a good example of this type of application service.
Cloud-based Client Services
Think of a device that is completely useless without a connection to internet services (like the iPhone) and you will have an understanding of what
cloud-based client services are all about. Net books are another emerging computing approach predicated on utilizing more cloud-based
applications and services using a smaller, less powerful laptop computer.
Data Storage, Duplication and Backup
There are a number of variations of data services available via the cloud. Simple data storage allows for a pay-as-you-increase sort of service
allowing ever increasing disk utilization as needed. Other services offer data duplication and mirroring, and data backup.

Cloud Testing- attracting demand

Business applications are becoming more dynamic, distributed and component-based. This is due to demand for applications that are easier to use, more
flexible and more connected to real-time data. Consequently, the task of testing and delivering these applications is rising in importance and criticality.
This is where cloud testing has emerged as a fresh approach to testing where cloud computing environments are leveraged to simulate real-world user
traffic by significantly decreasing costs. This can also be extended to classical functional, regression and other testing of regular products in a product
development cycle from a perspective of cost. In essence, cloud testing is a form of software testing wherein testing is done using resources, machines or
servers from the cloud infrastructure. Besides, the entire testing environments can be obtained from the cloud on-demand at a cost that is practical and
reasonable due to the pay-for-use nature of cloud computing and with a lead-time that is near impossible within a company’s own data center.
Initially, this concept took shape when companies started using numerous machines booted up in the cloud in order to simulate web traffic and carry out
performance tests on Web sites. Now remote machines in the cloud are used to provide a common ground for testers to test and developers to isolate and
resolve the observed software defects. Apparently, cloud testing has traditionally been used to refer to load and performance testing of Web sites. However,
with increasing maturity of technology, all kinds of enterprise software can be tested for functional and performance issues before going in for full fledged
enterprise deployment.
“Cloud testing is the answer to the less-than-realistic performance test that originates within the infrastructure of one of our clients. When we use cloud
testing, we take advantage of hardware and bandwidth that more closely mimics our observed, real world conditions. Essentially, we execute the test in
cloud-based infrastructure and bandwidth. This allows us to design and execute global tests that give more meaningful results much faster.”
“Cloud testing basically aligns with the concept of cloud and SaaS. It provides the ability to test by leveraging the cloud, thereby bringing the same benefits
that the cloud brings to customers. With cloud testing, we don’t have to procure perpetual licenses for testing, or set up an expensive infrastructure to test
the applications. Here, we can leverage the shared environment provided by the cloud testing vendor,”
Of late, cloud testing is being offered as a service by some providers, which in itself is a new line of business. This service enables the users of cloud
testing to reduce or optimize on cost while having the ability to extend testing functionality.
Development of cloud testing
Cloud computing has inherited challenges in terms of security, reliability, and manageability. Mitigating these challenges has lead to the development of
cloud testing. Industry experts are of the view that cloud computing does have its share of challenges. It is definitely more secure and easy to manage if any
company has on-premise infrastructure. Having put any important data out in the cloud would only require the companies to be doubly sure about the
working of their applications and security of their data. In order to achieve that, it is a must to rigorously test applications using various approaches
including cloud testing.
“In the last couple of years, even though the cloud has raised the bar in terms of the infrastructure management, the above challenges are present in the
market. Hence, a proper testing cycle will definitely ensure that these are addressed at the right time in the product development lifecycle and deployment.”
Here, we would also like to mention that with the increasing scope for cloud computing, there is a spike in demand for cloud testing as well. Cloud
computing is growing at a rapid pace. With the advent of this technology, there is bound to be an increase in demand for cloud testing.
Moreover, the hype and popularity around cloud computing has certainly increased the visibility of cloud testing. Many virtues of cloud computing such as
scalability, dynamic allocation of resources and the pay-per-use model, which extends to cloud testing, make it an appealing option for enterprise
customers.
“As more applications move into the cloud, we will need to ensure that they still provide the same level of service as when run out of dedicated, on-premise
infrastructure. For example, an application running in the cloud would need to be tested for network, server, database and software performance as well as
the way in which it is cached in the client and security. This means testing would need to be done in the cloud and using the cloud, in turn, pushing the
demand for cloud testing,”
Practical advantages
As testing in the cloud is an on-demand service, scale-up or scale-down provisioning is much faster, turnkey and several times cheaper for organizations
with evolving application needs and fluctuating demand for IT infrastructure. Among other practical advantages, we could clearly say that cloud-based
tests are much more versatile. They can easily handle 10,000, 20,000, or even 100,000 users from a virtually infinite number of generation sites around the
world. The trick is to pick the sites that best mimic your expected Internet traffic. Secondly, they are much less expensive. They are executed on cloud
environments and use cloud bandwidth. This means that clients do not have to make large investments in infrastructure or connectivity. The software used
is also deployed in a SAAS model, so it is effectively priced and customers merely pay as they go. Finally, these tests are very fast to setup and execute.
“An ideal testing environment should mimic the actual production environment (including the hardware) which becomes expensive. Cloud testing helps do
away with this. Software testing requires realistic bandwidth to simulate this, which is not always available within the enterprise. Load balancing is a
critical parameter that needs to be tested and is at risk of being compromised if adequate IT infrastructure is not available,” One area where we would
probably see a practical advantage is where Web-based applications are involved. The ability to test a Web application at a particular scale at the level at
which it is expected to perform is a great advantage. Such infrastructure most probably will not be available internally to a corporation.
The principal advantages here are the ability to obtain computing resources from the cloud as-and-when needed to perform testing for a fraction of the cost
and in a fraction of the lead-time that it would take to purchase and maintain those computing resources within an organization.
Indian market scenario
The Indian growth story in this sector is going to be enabled by IT. IT use and implementation is still restricted to large corporations in India; cloud
computing and specifically testing in the cloud will bring about the required efficiencies and operating model for large scale IT implementation and
utilization in India. According to Springboard Research, the Indian market will see a gradual uptake of testing in the cloud as awareness spreads. With
software adoption steadily on the rise, cloud testing has good potential in the Indian market.
As more applications are converted or developed on the Internet, industry experts feel that cloud testing will become the fastest growing performance test
method. Moreover, customer adoption for SaaS-based applications in India is an area to which attention must be paid. With the adoption of cloud based
applications on the rise, cloud testing is also expected to grow. As more independent software companies in India develop SaaS applications, it will further
fuel this segment.

Cloud Based Open Source Testing Tools

 Ubuntu Enterprise Cloud 10.10 Alpha 2

 openQRM
 cloudtools
 TestMaker
 SOASTA CloudTest
Cloud Testing – The RoI
Cloud based testing approach can significantly drive the business capabilities of the Enterprise and its ability to place the
Product/Application in Cloud based Market Place, time to market in addition to other attributes such as help us Reduction in
Cost, Reduction in Risk, Increase in Productivity, Increase in Quality and finally it gives Customer satisfaction.

 Saving time and money

 Improving collaboration and coordination
 Improving speed and overall quality of testing & development
 Meeting service levels of non-IT workers
 Reapply time and resources to focus on the core business priorities
Conclusion –
Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user
experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for
users of many walks of life, exacting software testing standards must be met. Cloud testing with its many advantages such as cost-
effectiveness and scalability (giving access to tens and hundreds of required machines) is a strong enabler to maintain competitiveness in the software
testing market and companies will therefore increasingly embrace cloud testing.

References –
• Fundamental Requirements of Successful Testing in the Cloud
• Computing in the Clouds
• Cloud computing shapes up as big trend for 2009
• Cloud testing White Paper
• Cloud Testing - Wiki
• Cloud Testing Blog
Author’s Biography –
This is Sailaxmi R, have around 8+ yrs of Software development & testing experience. Spent almost 6+ years in CA HYD
INDIA, as a Principal Quality Assurance Engineer and lead couple of Cloud based testing teams and initiatives. Currently,
working with Cura Software, HYD on the Cloud based services platforms.

Sailaxmi has deep expertise in both Product development & testing using various tools. Work ed on the Network related
products like system management and application management etc. Mainly focused on the manual testing, performance and
scalability testing using gird technology etc

Appendix
ACM – Application Configuration Manager
SaaS – Software as Service
EC 2 – Elastic Computing Cloud