2011

Practical Guide for Choosing the Right Mobile Testing Solution for Your Enterprise

MOBILE MARKET OVERVIEW

The mobile boom presents tremendous business opportunities for enterprises. With a total of over 300 million smartphones shipped in 2011 and with mobile app store sales exceeding $6 billion, the market for mobile applications is ready for prime time. Those enterprises able to get to market quickly with highquality mobile apps will reap the rewards.

Figure 1 - Mobile phone market dynamics: Android, iOS, Windows and BlackBerry releases and new devices in a single year

Perfecto Mobile | Mobile Market Overview

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

As more and more enterprises "go mobile", they are discovering that traditional software testing methods are no longer suitable to ensure the quality and performance of their new mobile apps and websites. In today's extremely fragmented and dynamic mobile market, there is a need for a new testing approach designed to address the unique needs and challenges of the mobile environment. Choosing the right mobile testing solution is a key criterion for success as enterprises roll out their mobile business strategies. Perhaps the easiest way to underline the challenge of mobile application testing is to consider this staggering fact: over 260 Android devices have been released since the beginning of 2011. New smartphones and tablets (a segment which didn't even exist two years ago!) are released to the market on a weekly basis, each with a unique form factor or set of features. With three major operating systems on the market and an ever-growing number of versions, models, screen sizes and network technologies, it's no surprise that enterprises find it virtually impossible to keep their mobile apps and services in sync with ever-changing market trends. How does all of this affect the way enterprises should conduct their mobile testing? What are the implications of today's fragmented, dynamic and localized mobile environment on your mobile QA strategy? When you release a mobile app (or website), what do you need to consider? This whitepaper provides practical "how to" information that will assist enterprises in choosing the right mobile testing solution. It will help you to understand how mobile testing is different than traditional software testing, as well as outlining key criteria that enterprises should consider when selecting their solution. Lastly, we present a real world example of a mobile testing solution built to meet today's market challenges, along with the results achieved by an enterprise using this solution to test its mobile applications.

MOBILE TESTING IS MORE THAN SOFTWARE TESTING

Make no mistake mobile applications are software, and mobile testing is by definition a type of software testing. In other words, all your existing quality standards and ALM methodologies need to be applied in accordance with business objectives. In fact, a key guideline when going mobile is to change as little as possible in terms of ALM infrastructure and resources. But there's one catch. Mobile testing involves added complexity due to several new factors inherent in mobile that do not exist in traditional software QA. When choosing a mobile testing solution you will need to take these factors into account: Multiple platforms and networks As mentioned above, mobile applications must be tested on at least three major operating systems (i.e., iOS, Blackberry, Android). Each OS has multiple versions and Android, for example, rolls out a major release every 6-9 months (Honeycomb, Ice Cream Sandwich etc.). Device types, form factors and network technologies also need to be taken into account. Device logistics , procurement and security Enterprises require solutions that minimize the hassles and logistics associated with procuring, deploying and maintaining dozens of devices in multiple geolocations. Solutions that facilitate the logistics of offshoring mobile testing (shipments, network availability, regulations, etc.) can save substantial time and money. In addition, the security aspect of

Perfecto Mobile | Mobile Testing Is More than Software Testing

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

those mobile devices is critical - you must make sure those new "testing devices" do not breach corporate security policy. Collaboration among globally distributed teams Testing might be performed outside the target region of the application, i.e., off-shore testers require access to mobile devices from abroad. Globally distributed teams require real-time collaboration tools that enable a tester to share a device in "live" testing with one or more distant colleagues for their input or approval. If, for example, your mobile testing is offshored to India, your R&D team is in Eastern Europe and your target market is in the US, you need a solution that enables handsets to be tested and shared among these teams. Adapting to rapid changes "Here today, gone tomorrow" is probably the best way to characterize the pace of change in the mobile market. Approximately 30% of handsets are replaced every quarter. These extreme market dynamics dictate the need for continuous mobile testing to ensure that your applications remain relevant to the devices being used in your market.

Figure 2 Android devices compatible with specific Android versions (Source: Android Developers)

Perfecto Mobile | Mobile Testing Is More than Software Testing

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

UNDERSTANDING THE MOBILE DEVELOPMENT CYCLE

The illustrations below depict the various testing activities required throughout the mobile application lifecycle. While many of these activities correspond to the classic software development model, the unique characteristics of the mobile market impact each and every one of these activities. Moreover, there are aspects of mobile testing (e.g., interoperability, porting) related to platforms and networks that are unique to the mobile application lifecycle (depicted in orange) as explained in the next sections.

Figure 3 - Testing activities by phase, software QA vs. mobile QA (simplified)

Perfecto Mobile | Understanding the Mobile Development Cycle

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

Think Platforms Know Your Market In order to address market dynamics, mobile applications need to be developed and tested on multiple platforms. You cannot assume that an application developed on Android will run smoothly on iOS. Thus, it is important to identify 4-6 "must" devices and run rigorous nightly sanity testing and regression testing on these devices. During the QA phase, it is recommended to extend the testing to cover ~10 "major" devices to achieve a better representation of the market. The bulk of the functional and regression testing would be performed against these devices. Automation in both of these phases is critical in order to allow the release of new apps and functionality to the market in a timely fashion. Regardless of the phase, around 30% of the devices will need to be replaced each quarter to account for new devices being introduced to the market.

Figure 4 "Must", "Major" and "Market" devices used on different testing stages (simplified)

Think Connected - Interoperability Since the mobile device is a connected device, its interface with the environment is different than the average web or desktop application. Network-related events could affect the quality of your application. What happens, for instance, when a call or text message comes into your device in the middle of a transaction? Different scenarios must be tested since a slight variation in timing could have a major effect on the results. In addition, different network technologies (e.g., HSPDA, WCDMA, UMTS, LTE) might change the way your application behaves and must be taken into account.

Perfecto Mobile | Understanding the Mobile Development Cycle

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

Automation can greatly reduce the risk involved in interoperability testing. While eliminating the errors that inevitably result from manual testing, an automated system can run these tests systematically and record the details of each iteration. Continuous Testing - Compatibility Compatibility testing (porting) requires continuous testing on an ever-changing stream of new devices and tablets being introduced to the market. Compatibility testing is aimed at making sure your application operates properly on the 30-40 most relevant devices in your market. This stage is more conducive to manual testing since the testing will be carried out on multiple carriers, form factors and OS versions. The most important aspect of this testing is to keep your selection of "market" devices up to date by replacing ~10 of these devices every three months. This means that you will need a logistical solution which will enable you to ensure availability of these devices when you need them, as well as collaboration between testers and R&D. Consider the security aspects as well, since "loose" management of these devices could inadvertently lead to IT security breaches. Examples include enabling devices to be in unknown locations, giving them access to your internal network, or allowing access to your USB ports (some corporate policies disable this possibility by default to prevent information leakage). Shorten Development Cycle to Minimize Requirement Gaps In traditional software development, new software versions are released to meet ongoing market requirements. During the time between releases, the gap between the software functionality and the market requirements begins to grow until the next version closes the gap. In the mobile world, due to the extreme dynamics and new technologies being introduced to the market, the gaps grow much faster than in the standard software world. To address this issue, it is very important to release software versions as frequently as possible, avoiding large gaps and allowing for applications that successfully address the needs of users.

Risk
Gap from requirements
Mobile Dev.

Short Cycle Mobile dev.

Time
SW Cycles

Figure 5 - Shorter mobile development cycles reduce risk, every release closes the gap from existing requirements

Perfecto Mobile | Understanding the Mobile Development Cycle

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

SELECTION CRITERIA FOR YOUR ENTERPRISE MOBILE TESTING SOLUTION

With an understanding of the particular business and technological challenges of the mobile environment, as well as the key testing activities in the mobile development cycle, you are now ready to select the right mobile testing solution for your enterprise. This section outlines the key selection criteria that you should consider. Automation is Imperative As development cycles become shorter over time, the need for automation in regression testing has turned from nice to have into mandatory, particularly when using agile or iterative development methodologies. By automating the functional and regression testing of your mobile applications, you can ensure the quality and performance of your applications on any device, OS, network and location. Automation allows more tests to be performed on more devices in less time and reduces the "requirements gap." The result is a shortened and systematic ALM cycle that allows for continuous QA, better coverage, easier re-creation of problems and substantial cost savings. Be careful not to underestimate the complexity of launching mobile apps. It is common to experience a cycle of over-optimism following development on the first platform (OS), followed by disillusionment when difficulties are encountered once the second platform and associated devices are added to the mix. The realities of an extremely dynamic market require a well-planned and methodical approach. In light of this fragmentation, it is highly recommended to adopt a device-agnostic testing approach that allows you to write test scripts once and then re-use them on multiple platforms. Script automation should support low level functionality (e.g., touching a key, touching the screen) as well as logical abstraction which enables the execution of virtual functions (e.g., login) that are not dependent on a particular device or platform. Cloud-Based Access to Devices To ensure application compatibility on the platforms that matter to your users, best practices indicate that you'll need access to 30-40 fully functional devices at any given time, and you'll have to replace approximately 10 of these devices each quarter to stay up to date. The logistics of managing this number of handsets in diverse geo-locations is challenging, to say the least. The amount of supported devices should be considered. With the rate of introduction of new devices and OS versions, assume that for the first year the amount of devices will grow significantly as your user base builds legacy support. Cloud-based access to REAL handsets located in live networks allows you to avoid the hassle and costs of procuring and managing new devices (i.e., shipments, network availability, active plans, SIMs, etc.). Since all devices are managed in the cloud, you no longer have to worry about complicated logistics. Devices can be easily added or replaced within a few hours in response to market needs. In addition, a cloudbased approach facilitates offshoring and enables globally distributed teams to share a device in "live" testing and to replicate issues for improved troubleshooting.

Perfecto Mobile | Selection Criteria for Your Enterprise Mobile Testing Solution

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

The table below presents a comparison of cloud-based mobile testing with other types of solutions:

Figure 6 - Comparison: Different approaches to mobile testing

Use Existing ALM Resources For enterprises with an ALM infrastructure already in place, the goal is to minimize the cost of new tools and skill sets required for mobile testing. Rather than creating a new separate ALM environment, you can leverage existing tools and processes by extending your current ALM framework to support mobile testing. By managing mobile, desktop and web application testing through a single integrated ALM platform, organizations can centrally manage and track all application projects and ensure consistent workflows and processes. This will help to eliminate the costs associated with maintaining, managing and synchronizing separate systems for desktop and mobile testing. The integration of mobile testing within your existing ALM enables centralized and complete test management and visibility for your mobile apps. Using a single platform for managing and automating the application lifecycle helps to ensure mobile quality and performance while reducing total cost of ownership. Natural Extension of Your ALM Platform As opposed to a simple exchange of data items between two disparate applications, creating a natural extension of your ALM platform to mobile will ensure a seamless user experience. By truly embedding the mobile testing capabilities within your ALM environment (e.g., HP ALM, IBM, etc.), testers can work within the existing application, manipulating mobile functions from within the system using known language and syntax. Mobile testing solutions that use a familiar user interface, scripting language and workflows will reduce learning curves and allow for a rapid implementation. This enables you to maintain the same functions and staff you already have in the organization, while preserving the knowledge and experience of your team. Perfecto Mobile | Selection Criteria for Your Enterprise Mobile Testing Solution 8

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

Secure Environment To meet your enterprise's security requirements, it is essential that mobile application testing can be performed on a dedicated bank of handsets (i.e., private cloud) that can only be accessed by your organization. This ensures that the handsets you need are always available for testing, and that your application and data are always secure. The private cloud should be configurable to comply with your organization's security policies, including firewall requirements and other needs.

CASE STUDY: EXTENDING HP AUTOMATED TESTING TO MOBILE

Perfecto Mobile and HP have collaborated to develop a complete and fully integrated solution for automated testing of mobile applications within QuickTest Professional (QTP). Already deployed by a number of enterprises, MobileCloud for QTP integrates the core scripting language and flow control of QTP with Perfecto Mobile's comprehensive cloud-based automated mobile testing solution. This proven solution enables enterprises to naturally extend their existing HP Functional Testing and ALM environment to support mobile applications.

HOW IT WORKS
MobileCloud for QTP gives testers immediate access to hundreds of devices located anywhere in the world from inside QTP. To create a mobile test script, testers login to QTP, connect to the MobileCloud and choose one or more devices to work on. Users work on the device directly from the familiar QTP user interface, using the rich feature set and scripting environment provided by QTP. There is no need to learn any additional applications. Tight integration at the UI level allows testers to use the same UI for normal QTP scripts, mobile scripting and hybrid scripting. MobileCloud for QTP enables high-level intelligent mobile specific functions that mimic the actions that a real user would take when operating a mobile device (e.g., press keys, open browser, select image, type text, etc.), allowing fast, easy and maintainable mobile scripting. Perfecto Mobile's ScriptOnce technology enables advanced functions, such as dynamic search for text and images (including scrolling) on screen and visual relationship association that facilitate the creation of device-agnostic scripts during the recording phase. Complete and detailed testing information from executed mobile commands are embedded directly into the QTP report. These enhanced QTP reports include active screenshots and video recording of the test run, as well as standard information such as test status and description. The MobileCloud for QTP solution enables enterprises to cost-effectively extend their functional and regression testing to mobile through the following capabilities: Automated mobile testing that leverages your existing QA knowledge, platforms and workflows Secure cloud-based access to hundreds of real devices in live networks Device-agnostic mobile scripting enabling re-use and porting of scripts to multiple platforms 9

Perfecto Mobile | Case Study: Extending HP Automated Testing to Mobile

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

Native QTP scripting for mobile eases script creation and maintenance Rich QTP reports with embedded screenshots and video improve collaboration among teams Full back-end integration with HP Quality Center enables centralized test management

Figure 7- HP QuickTest Professional UI with MobileCloud integration

REAL-WORLD BENEFITS
MobileCloud for QTP was implemented at a large financial institution to test mobile applications (native, mobile web and web) that process financial transactions on a variety of iPhone, Android and BlackBerry devices. One of the test scenarios called for the transaction to begin on a mobile device and then verify it on the website. This complex scenario demonstrates the unique value of the integrated solution, i.e., the ability to leverage existing QTP assets and to call mobile assets as needed directly from QTP. The financial institution achieved the following tangible results using the MobileCloud for QTP solution: Reduction in Mobile regression testing time From 3 weeks to 2 days to execute more than 30 end-to-end test cases test on more than 10 handsets Improved Device Coverage - Ability to change and support new devices for automated testing in a "plug and play" manner as needed. Improved Test Coverage Broadened testing to 30+ test cases across 10+ handsets (compared to 4 handsets previously) Improved Quality QTP based automated enabled end-to-end testing for transactions across web and mobile systems Debugging support QA team can provide video of the defect to developers and provide access to remote device in private cloud for debugging and issue resolution Better QA Visibility for Management Automated mobile test results are available in HP Quality Center for consolidated reporting across platforms and end-to-end quality measurements 10

Perfecto Mobile | Case Study: Extending HP Automated Testing to Mobile

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

CONCLUSIONS
To address the complexities of mobile application testing in a highly fragmented and dynamic mobile market, enterprises need to implement a well-designed mobile test automation strategy. This strategy should reflect the unique needs and challenges of the mobile environment, as well as new aspects related to the mobile development cycle (e.g., interoperability, porting) that do not exist in standard software ALM. Unlike desktop application development, in the mobile world it is critical to think platforms. With over 580 different mobile devices released to the market in the first eleven months of 2011, mobile applications need to be developed and tested on multiple platforms and operating systems. Moreover, since new devices are constantly being released to the market, enterprises must continuously test their applications to ensure compatibility with the most relevant devices in their market. This is an ongoing process since one-third of these devices will need to be replaced each quarter to keep pace with market changes. Given the extreme market dynamics, shortening the mobile ALM cycle can help to avoid large gaps between versions and market requirements, allowing for applications that adapt to the changing needs of your users. As mobile apps and services become business-critical, your choice of a mobile testing solution will have a major impact on business operations as well as your organization's internal QA activities. When choosing your solution, it is recommended to consider the following main selection criteria (dubbed "ACE"):

Automation mobile test automation will enable you to shorten the ALM cycle, increase
coverage, facilitate re-creation of problems and save costs

Cloud-based platform - Cloud-based access to REAL handsets located in live networks allows you
to avoid the hassle and costs of procuring and managing new devices, while facilitating offshoring and collaboration

Use Existing ALM Resources - Leverage existing tools, processes and knowledge by extending your current ALM framework to support mobile testing

Solutions that meet these criteria will help your enterprise to address the complexities of multi-platform development in a highly fragmented mobile market, while enabling you to cost-effectively extend your ALM environment to mobile.

ABOUT PERFECTO MOBILE

Perfecto Mobile is a global leading provider of cloud-based testing, automation and monitoring solutions for mobile applications and websites, utilizing a wide selection of REAL and emulated mobile devices accessible via the web. Perfecto Mobiles MobileCloud Platform enables developers and testers to access and control a comprehensive range of the latest mobile devices connected to live networks around the world. Users can rely on Perfecto Mobiles handsets and tablets to develop, test, deploy and monitor their mobile applications and services without having to physically obtain them.

Perfecto Mobile | Conclusions

11

Practical Guide for Choosing the Right Mobile Testing

December 1, 2011

Perfecto Mobile and HP have collaborated to develop the world's first complete solution for automated testing of mobile applications within HP's QuickTest Professional (QTP) product. MobileCloud for QTP integrates the core scripting language and flow control of QTP with Perfecto Mobile's comprehensive cloud-based automated mobile testing solution. This ground-breaking solution enables enterprises to naturally extend their existing HP Functional Testing and Application Lifecycle Management (ALM) environment to support mobile applications with both real devices and emulators.

Perfecto Mobile | About Perfecto Mobile

12