Testing Mobile Applications Using Perfecto Mobile MobileCloud V2

ALM Rangers - Testing mobile applications with Perfecto MobileCloud
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT. Microsoft grants you a license to this document under the terms of the Creative Commons Attribution 3.0 License . All other rights are reserved.
2013 Microsoft Corporation.

Microsoft, Active Directory, Excel, Internet Explorer, SQL Server, Visual Studio, and Windows are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners.
Page 2 of 49
Table of Contents
Introduction ...................................................................................................................................................................................................................................... 4 Overview ....................................................................................................................................................................................................................................... 4 Visual Studio ALM Rangers ................................................................................................................................................................................................... 4 Contributors ................................................................................................................................................................................................................................ 4 Personas ........................................................................................................................................................................................................................................ 4 Related Resources..................................................................................................................................................................................................................... 4 Exercises ........................................................................................................................................................................................................................................ 5 Device testing approach for IT Applications ....................................................................................................................................................................... 6 Need for Device Testing ......................................................................................................................................................................................................... 6 Challenges in Device Testing ..................................................................................................................................................................................................... 7 Device Testing Lifecycle for IT Applications (DTL-IT) ................................................................................................................................................. 7 Envision Phase ............................................................................................................................................................................................................................ 7 Plan Phase .................................................................................................................................................................................................................................... 8 Build Phase................................................................................................................................................................................................................................ 11 Stabilize Phase ......................................................................................................................................................................................................................... 14 Native App Best Practices ................................................................................................................................................................................................... 14 Environmental preparation ...................................................................................................................................................................................................... 16 Installing the Perfecto Mobile Team Foundation Server (TFS) add-in............................................................................................................. 16 Accessing the Perfecto Mobile Cloud Platform ........................................................................................................................................................ 17 Hands-On Lab ............................................................................................................................................................................................................................... 20 Exercise 1 Accessing the Perfecto Mobile Cloud and running an automated test (10 min) .............................................................. 20 Exercise 2 Run an automated test in the Perfecto Mobile Cloud within Visual Studio (15 min) ...................................................... 26 Exercise 3 Run an automated test in the Perfecto Mobile Cloud within Microsoft Test Manager (10 min) ................................ 31 Exercise 4 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on TFS (10 min) .............................................................................................................................................................................................................................................. 38 Exercise 5 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on Team Foundation Service in the cloud (10 min) .................................................................................................................................................................... 41 Conclusion ...................................................................................................................................................................................................................................... 45 Appendix ......................................................................................................................................................................................................................................... 46 Automation Tool Comparison .......................................................................................................................................................................................... 46 FAQ .............................................................................................................................................................................................................................................. 48
Page 3 of 49
Introduction
Overview
How do you develop and test apps on iOS, Android, and Windows Phone devices? You leverage the Visual Studio Industry Partner (VSIP) solutions and our Team Foundation Server (TFS) integrations. For example, for IOS and Android development, use solutions from ITRMobility and Xamarin to develop within the Visual Studio IDE using .NET. You can use Xcode, Eclipse, or IntelliJ and integrate with Team Foundation. Then you test your mobile application using Perfectos Mobile Cloud solution. Mobile applications can require complex testing on specific devices for which the acquisition and maintenance can be expensive. Especially for devices based on Android as there are so many variations. Emulators are useful but can be expensive and inaccurate. The best solution is to test your mobile applications on the actual device using Visual Studio Application Lifecycle Management (VS ALM) and TFS tooling enabling an end-to-end Application Lifecycle Management (ALM) experience. Deploying your application on the actual device, testing from anywhere globally, and integrating with Visual Studio and TFS provides incredible business value. The following guidance will describe how to install and setup Visual Studio and TFS to use the Perfecto Mobile MobileCloud for TFS solution. The document also describes how to use the Perfecto Mobile solution within the Visual Studio 2012 RTM ALM Virtual Machine or your own VM or local desktop. In addition, we will discuss the architecture, the mobile cloud, testing of real devices, ScriptOnce, the integration with Visual Studio and Microsoft Test Manager, and the end-to-end scenario.
Visual Studio ALM Rangers

The Visual Studio ALM Rangers are a special group with members from the Visual Studio Product group, Microsoft Services, Microsoft Most Valuable Professionals (MVP) and Visual Studio Community Leads. Their mission is to provide out-of-band solutions to missing features and guidance. A growing Rangers Index is available online1.
Contributors
Anil Chandra Lingam, Anisha Pindoria, Brian Blackman, Dan Marzolini, Esteban Garcia, Hassan Fadili, Mathew Aniyan, Oliver Hilgers, Bipin Puthiyaveetil, and Ranjit Gupta.
Refer to Visual Studio ALM Rangers Personas and Scenarios 2 for more information about the personas and customers profiles referenced in this guide.
Related Resources
Understanding the ALM Rangers http://aka.ms/vsarunderstand Visual Studio ALM Rangers Home Page http://aka.ms/vsarmsdn Visual Studio ALM Ranger Solutions http://aka.ms/vsarsolutions Visual Studio ALM Ranger Solution vsarDeviceTest Project on Codeplex - http://aka.ms/vsardevicetest
1 2
http://aka.ms/vsarindex http://go.microsoft.com/fwlink/?LinkID=230942
Page 4 of 49
Exercises
The Hands-on lab sections will show you how to use the Perfecto Mobile platform and Visual Studio to test different mobile devices through automated scripts as a manual or general tester. The lab will take approximately 55 minutes to complete. The hands-on lab is structured for you to learn how to: 1. 2. 3. 4. 5. Run an automated test script on the perfecto mobile cloud and review the test results. Create your own Visual Studio Perfecto Mobile Cloud project and run tests against real mobile devices in the cloud. Run an automated test script against real mobile devices in the cloud within Microsoft Test Manager 2012. Run an automated test script within a Continuous Integration process on TFS Team Build. Run an automated test script within a Continuous Integration process on Team Foundation Service Build in the cloud.
This guide targets the Microsoft 200-300 level users of Visual Studio 2012. The target group is considered as intermediate to advanced users of Visual Studio 2012 and has in-depth understanding of the product features in a real-world environment. Parts of this guide may be useful to the Visual Studio novices and experts but users at these skill levels are not the focus of this content.
WARNING
The mobile device environment might change. Please read the next paragraph carefully.
The exercises run on real devices in the Perfecto Mobile Cloud. As real mobile devices are subjected to software updates and other environmental changes over time, it is possible that tests could on occasion fail. Therefore, if a test execution in the exercises fails the root cause can be different. Here are some examples:
Possible error scenario The URL https://demo.perfectomo bile.com is not available anymore. The test fails with an error due to a device error. Possible root cause The demo environment is not available anymore and the production environment can be used in a trial mode. The referenced mobile device in the exercise doesnt exist anymore. is disconnected. has a new device ID. is locked or blocked by another user. Solution Use another URL in the exercises and Visual Studio Code: Use the production environment URL: https://www.perfectomobile.com. Validate the mobile device status: Please check the portal https://demo.perfectomobile.com in the area Interactive / Select Device and validate the device status. If the needed device has an issue use another available device. For the exercise the device ID (DUT), device name and OS name (os) are needed and must be changed. Then retry the exercise / test execution. Validate the app status on the mobile device: If the ATM app doesnt exist on the mobile device please contact the Perfecto Mobile support. Validate the mobile device screen: Close the pop up message without executing the OS device update and retry the exercise / test execution.
The test fails with an error due to an app error. The test fails with an error due to an OS update pop up on the device start screen.
The ATM app is not installed on the selected device. A pop up message on the device shows that an OS update is available. So, the tests script cannot select the ATM app icon anymore.
Table 1 Possible test execution error scenario.
For additional help please contact the Perfecto Mobile support: http://www.perfectomobile.com.
Page 5 of 49
Device testing approach for IT Applications

Need for Device Testing
The emergence of smartphones and tablets has changed the dynamics of internet usage. As per the latest market research, over 40% of the users are accessing internet applications using devices other than PC. Success of the web applications is now heavily dependent on the quality of service (QoS) on both PC and devices. We are now observing similar patterns with regard to enterprise applications. The culture of BYOD (bring your own device) is being welcomed by several major enterprises worldwide. This has led to interesting challenges and opportunities, one of them is huge demand for Device testing, for ensuring consistent experience across devices. Given the dynamics of the IT applications usage, testing approach must also undergo surgical changes. In this paper, we highlight the changes needed to testing approach and present the approach for device testing.
Figure 1 - PC verses Devices Spread across verticals
Page 6 of 49
Challenges in Device Testing

There are now over 2000+ types of devices in the market and it is not feasible from both cost and time perspective to test IT applications on every device type. This presents a significant challenge of identifying a minimum set of devices which would represent the majority of the devices. Also, depending on the application usage, locale and end-user persona, the minimum set for testing of devices can vary. For example, if a company in Australia wants to adopt BYOD culture, then minimum set of devices on which an enterprise IT application needs to be device tested would depend on the devices widely used by the employees in Australia. Hence, identifying the minimum set of devices is key for successful device testing. There will be a single testing strategy for PC, mobile and tablets, which optimizes the testing cycle from cost, quality and time perspective. There is another school of thought that considers strategy for traditional testing on PC and testing on devices should be separate. While there are some advantages of such approach, we have observed that, a single testing strategy for both traditional testing on PC and testing on multiple devices is cost efficacious. Specially, we have observed following advantages: Reducing the duplication of test cases Reuse of test data Effective sharing of test resources and test artefacts
Device Testing Lifecycle for IT Applications (DTL-IT)

Envison Obtain Customer requirements specific to usage of the application on different devices Obtain customer requirements on the device spread Obtain NFR Plan Build Create device automation framework Create Test Cases Stabilize Execution of test cases on emulators/real devices based on the test spread Run the Tests Publish the Test Report
Define scope Conduct Automation Assesement/ROI Define test spread based on schedule Create Device Testing Strategy Define Test Sceanrios
Figure 2 - Device testing life cycle for it applications
Test Spread: Define the % effort in emulators, manual and cloud-based testing.
Envision Phase
How to select the minimum set of devices to be device tested?
Selecting the minimum set of devices that needs to be device tested is the most challenging aspect of device testing. This would depend on factors such as platform to be supported, business usage of the application and market data consideration for device market. For a typical web application that will be consumed on multiple devices would have the following device priority matrix:
Page 7 of 49

Priority High Priority Platform (OS) Browser (IE, Firefox, Chrome, Safari, etc.) Form Factor (7'', 10'', 4'', etc.) Medium Priority Input Mechanism (Stylus, Swipe, Camera, GPS, etc.) Network Connection (GSM,CDMA,BES* -BB, Edge,2G,3G,4G,Wi-FI) Low Priority Memory CPU Type (ARM, Intel, etc.)
Table 2 Factors to be considered for device test mix
Depending on the usage of your application, define the device priority matrix and arrive at a set of devices on which testing needs to be performed. Make sure that, you are choosing only those devices that are critical to your business. This activity needs to be done during the planning phase. Also, given the dynamic nature of device market, if time period between planning phase and SIT phase is significant, then re-evaluate the device priority matrix on every iteration, so that you are including the new devices that have just arrived to the market in your test mix. Another strategy for testing the application on multiple devices is to do Beta Release of the application and do dog-fooding.
Plan Phase
Figure 3 - Testing types across various layers for optimizing the efforts
Following are the key activities that needs to be performed during Plan phase: Define User Scenarios for the IT application
Page 8 of 49

Gather the cross platform / browser mix requirements of the scenarios and personas accessing those scenarios which will mimic the real life usage profile mix of the application. o Ex: 20 % of the total scenarios are accessed through mobile, 10 % through tablets and rest 70% over desktop/laptop o Mobile Spread is further split across Safari on IPhone 5 (60%) ,Android 4.0 /Chrome (35%), Widows 8/ IE10 (15%) o Desktop spread across Windows 8/Chrome, Windows 8 /Safari, Windows 8 /IE 10 Identify the scenarios that are critical to run on real devices like must /should /nice to run which will help, propose the right mix to the customer. Identify the device/browser spread across the customer user base if the customer does not have this information, get the top N devices that are widely used in the region to help the customer to make the right choice Plan for any upcoming device/browser launches and make provision to accommodate these mix to support agility Scope the devices/browser mix for the project based on the schedule /availability of devices and budget of the project. Scope the test spread mix of devices /emulators/browsers ideal for the engagement based on the above requirements Calculate the ROI based on coverage (device spread) versus risk of not testing across all spread Prepare the test strategy based on the above
Device support strategy 1) How many of the platform and devices should we support? 2) Business Considerations 3) Market data considerations 4)Failing gracefully when a device is not supported
Device Configuration Matrix

Device Model Input mechanism (Eg:Touch,Key) Screen Resolution /Orientation Chipsets/Peripherals (GPS,Camera,Microphone) Carrier Locked /Un Locked Upgrade/Install type (ROM.,OTA, App store, Tethered-kiosk scenario) Network Type (GSM,CDMA,BES* -BB) Network Connection (Edge,2G,3G,4G,Wi-FI) Operating System (Windows/Android/Blackberry) Browser
Recommendations
1) First Test on the major Desktop browsers to ensure that the system is functionally stable (VS 2012 Coded UI Selenium support will help for Automation) 2) Run the tests on major platform/devices Emulators (Automation/ Manual based on ROI) 3) If the devices are available then test on real device. Set of Reference devices in-house shared by the team. Test at least once all the scenarios in the reference devices manually
Page 9 of 49

4) Breadth device testing done by external party where the efforts are more .Needs to identify partners with this capability. 5) Cloud based mobile testing tools can be used in case the device spread is huge and one wants to automate once and run the tests across multiple devices over the cloud on a real device. This eliminates the maintenance overhead of managing the devices internally and optimizes costs by eliminating manual execution of tests.
Real /Virtual Mobile Testing Decision Table

Virtual Mobile Testing Tool Regression Test ,BVT Sanity Compatibility Large set of Functional Testing scope Wide Range of devices available for testing .Script once run on any device Devices UAT Compatibility Inter operability Interrupt Test Performance End to end scenario
Table 3 Mobile testing decision table
Based on the combinations identified one can make use of combinatorial algorithms to decide the optimum combinations of tests to be designed to ensure greater coverage and eliminate risks.
Device Testing Cost Pyramid
Figure 4 - Device testing Cost Pyramid
Emulators/Browser plug in based testing will be the most cost effective solution. The above pyramid helps make the decision based on the cost factor. This also depends on the number of devices in our test spread and the
Page 10 of 49

total number of tests to be executed. In case both variables are low then real device testing will work out to be more economical compared to cloud based testing.
Test Automation Tools for Device testing

Key requirements for the Device Test Automation Tool:
Should be able to port the automated scripts of one platform to multiple platforms and multiple devices .Script once run anywhere (Device anywhere/PerfectoMobile) Speed up test execution Eliminate manual efforts to reduce total cost Easy to maintain Should be extensible to support future platforms and devices with ease. Should be able to run the tests on emulators as well as devices on demand (Nice to Have on Devices as Running tests on real device takes lots of time and can be done only in serial fashion) Resilient and smooth execution of automated tests: This is very important as no one would like to have an automation that is not stable and needs frequent monitoring for failures. Automation framework should be able to give consistent and predictable results for every run.
Automation guidelines
Automation on Virtual Device is recommended for the following areas
1) Any scenario that can be run on an emulator can be automated and can be tested without the real device .Identify the regression suite for the functional test cases and we can automate those scenarios for all the gestures to ensure that the major risks and coverage issues are addressed as part of early iteration testing and regression tests are automated as part of iteration scope .Majority of the emulators comes free. 2) Automation on Real device is recommended in early stages of the application life cycle to detect and fix the bugs early. 3) Anything that can be run on a emulator for all the top 4 platform/devices 4) Performance and Stress testing on actual devices 5) Some Configuration testing on Emulators 6) Standard out of the box compliance check/design guidelines check
Automation is Not Recommended for the following areas

1) Configuration testing 2) UI look and feel verification testing 3) E2E Testing 4) Native App automation (Assuming not many complex functionality is delivered, Automation wont be recommended) 5) Scenarios which are already addressed through automation for desktop access 6) Usability testing
Build Phase
Following are the key activities that needs to be performed during Build Phase: Identify the test scenarios based on functional and non-functional requirements
Page 11 of 49

Make sure that, you are identifying User Scenarios that include gestures such as Touch Touch Zones Swipe Pinch Zoom
Identify the priority scenarios which can be automated based on the return on investment (ROI)
Useful tools for developers to catch bugs early

Tool Name Device Test Planning Description Helps plan the number of devices and platforms to be used in testing. The W3C mobileOK Checker is one of the best tools available to you, to help you test the efficacy of your Website on mobile devices. This tool performs several tests on a Web page before it estimates the level of compatibility of your Website with the mobile Web. W3C has developed a mobileOK Basic Tests 1.0 specification, which works to give you a clear idea of the mobile-friendliness of your Website. A very accurate iPhone tester, this is also available to you completely free of cost. Though iPhoney is not really a simulator, it enables you to build 320x480px Websites, compatible with the iPhone screen specifications. This implies that you can test both your code and your Website images in the actual Apple Safari type of environment, including all the features of the original iPhone, such as zoom, plugins, landscape and portrait modes and so on. Link Planning tool is here
W3C mobileOK Checker
W3c Mobile OK
Iphoney
Iphoney
Page 12 of 49

Gomez The Gomez mobile readiness test analyses your Website on the basis of over 30 wellestablished, fool proof, mobile Web development techniques. It then rates your page on a scale ranging between 1 and 5 points. This tool not only offers you more-or-less accurate results, but also offers advice on how you can further improve your site in order to make it even more compatible with mobile browsers. Make note of the fact that this tool first requires you to enter some personal information, before you can go ahead and use it. Gomez
Table 4 Device testing tools
Device Usability Testing

Different elements of the form factor influence the way the web application runs on the device. For example, the following factors are crucial when it comes to testing web application on modern devices: a. Screen Orientation The screen orientation Landscape / Portrait dictates how the application is shown on the browser. Scenarios must be identified which test the application in different orientations and look out for issues resulting from page overflowing from the screen, elements not being visible to the user etc. Also, if the application uses change in orientation as a trigger, that also needs to be exercised. b. Responsive design Modern web applications are adopting Responsive Design practices to retain homogenous look and feel across devices. The main objective of responsive design is to ensure that the same web application when opened across different devices of multiple form factors, provides similar user experience on all the devices, regardless of the screen size and resolution. This is accomplished by a combination of CSS and JavaScript which render the width and layout of elements in such a fashion that all elements are visible on the screen. c. Control Replacements There may be some controls on the web page which may be taking mouse or keyboard exclusive controls, for example mouse over events, mouse wheel events, pressing escape button etc. When the web application is run on devices, there should be alternative ways of triggering controls, for example - a drop down triggered by mouse over could be displayed as a select triggered by touch. All such controls need to be identified and be tested on devices.
Combinatorial Testing to generate effective number of scenarios

While testing for devices, we could potentially encounter large number of test combinations arising from multiple device form factors, screen resolutions, screen sizes, OS and browser types etc. It may not be practical to run all tests on all possible combinations. Hence, it is recommended to come up with a condensed subset of
Page 13 of 49

scenarios which provides maximum test coverage using Combinatorial Testing. Combinatorial test generation technique is based on the premises that defects occur in a system due to interaction between few numbers of components (empirically less than 6). If we find all possible combination of components taken 6 at a time, then we can uncover cover more than 90% of total number of bugs. There are various tools available which allow users to generate combinatorial scenarios; PICT, ACTS to name a few.
Security Testing on Devices

The following aspects of security needs to be tested on devices: 1. Authentication: Apart from the standard authentication methods the web application exposes, there could be device specific authentication methods like swipe gestures, Google account, Facebook account, device tokens, device certificates etc. Authorization: A few aspects of authorization that could be sensitive from devices perspective could be elevation of privileges, escaping License checks, insecure HTTP methods that could be tampered over network etc. Depending on the OS, there could be issues with file permissions on the local system Data Storage Web applications may use local storage for cache management, temporary downloads etc. Appropriate security testing needs to be done to ensure that this data does not contain any sensitive information and cannot be accessed by unauthorized parties. Generic Web application security Security test cases should be executed on the devices which exercise the application vulnerabilities which exploit the client side, for example: XSS and HTML Injection Command Injection (if the application utilizes a shell) CSRF SQL Injection Cookies HTML5
2.
3.
4.
Stabilize Phase
Following are the key activities that needs to be performed during the Stabilize Phase: Execution of test cases on emulators and real devices as per the Test Spread Run the Tests Publish the Test Report
Native App Best Practices

Native app testing and automation is no different than the webapp ,but we may have to script for each platform separately .We have some best practices for native application to ensure that native application that one build shines in the market place.
Page 14 of 49

Application start up Performance App should be responsive Indicate this using some progress bars Application Resumption- Calls, Messages, Back button Click- Recovery handling needs to be good for example in the middle of browsing if the user gets the call ,User should be shown the same content that he was browsing before the call. 3. Consistent User Experience Back button- First page back button should exit the app, for subsequent pages, it should show previous page. Dismissing the dialog should be supported by back button 4. Battery Drain Testing- Overall app should be benchmarked against some existing app in similar category for Battery drain. E.g.: Social networking app should be benchmarked against a Social networking app. Subsequent improvement can be done using battery drain tests. 5. Network handling Scenarios: 2G, 3G, Wi-Fi, Wi-Fi with proxies etc. For example: App should not hang when the network coverage is lost and should provide the right exception message back to the user. 6. Error Handling should be very meaningful. Not many mechanisms to troubleshoot issues. So, error messages should be very meaningful. 7. For apps where huge content needs to be added, recommend fetching it after installation. This will help the build to be smaller in size and help more market penetration as users wont download something which is bit heavy. 8. Application should not intervene into the platform features functioning. It should not override the platform capabilities and features. E.g. Location services setting of the app should not override the location services setting in the device 9. Application should not compromise on the security of data and other security features like XSS scripting 10. Landscape and Portrait mode should be supported 11. Application should work fine in different themes and background colors 1. 2.
Page 15 of 49
Environmental preparation
In this section, we will cover the pre-requisites and installation of the MobileCloud for TFS template for Visual Studio. Your options are to install the MobileCloud for TFS to your desktop where you have Visual Studio installed or to a Virtual Machine with Visual Studio. MobileCloud for TFS integration with unit testing does not require TFS. MobileCloud for TFS integration with Microsoft Test Manager does require TFS. The following guidance will at times seem like a hands on lab, however, some detail is needed as we explain what is needed to use the MobileCloud for TFS with Visual Studio and TFS. In order to run Perfecto Mobile scripts from your development environment, you first install the Perfecto Mobile Cloud for TFS Test Template. The installer creates a Project Template on C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ProjectTemplates\CSharp\Test and places Perfecto Mobile assemblies on C:\Program Files (x86)\PerfectoMobile\Visual Studio. This location is configurable during installation. Test projects created using the Mobile Cloud Project template use the installed TFS Adapter to make calls to the Perfecto Mobile service.
Installing the Perfecto Mobile Team Foundation Server (TFS) addin

Pre-Requisites

NOTE
Visual Studio 2012 + Update 2 or higher Microsoft Test Manager 2012 (required for integration with Test Case) TFS 2012 Update 2 (if on-premise deployment) or Team Foundation Service 3 in the cloud SQL Server 2008 / 2012 (if on-premise deployment) Perfecto Mobile Account - https://www.perfectomobile.com/trial/sign_up
A user account is required to use the MobileCloud for TFS. This guidance and the HOL exercises assume you have your own account.
For additional information, you may access the following resources: Microsoft Test Manager 2012: http://msdn.microsoft.com/en-us/library/vstudio/dd380763.aspx Team Foundation Server 2012: http://msdn.microsoft.com/en-us/vstudio/ff637362.aspx
Performing an Local Installation

Retrieve the installation package perfectomobile-tfs-add-in-installer-1.0.exe from https://www.perfectomobile.com/support/tfs and run the installer, confirming the User Account Dialog box if it is displayed. You will have the option to change the installation location folder if you wish. Once the Perfecto Mobile Installer completes detailed instructions on creating a test class & method to execute against the Perfecto Mobile cloud can be viewed here: http://aka.ms/perfectomobiletesttemplate
http://tfs.visualstudio.com
Page 16 of 49
Installing to a Hyper-V Virtual Machine

Similar, to a desktop deployment, the Perfecto Mobile TFS add-in template solution can be deployed to a virtual machine running in Hyper-V as long as an active network connection is provided to enable access to the Perfecto MobileCloud.
Installing to the Brian Keller Virtual Machine

If you are using Brian Kellers Visual Studio ALM Virtual Machine 4 to create and execute test scripts using the Perfecto Mobile TFS Add-in, additional configuration steps are required to set the current date and time and configure internet connectivity. All software pre-requisites are installed on the Brian Kellers Visual Studio ALM Virtual Machine.
Enabling Internet Connectivity

Perfecto Mobile is a cloud-based service so test cases must be executed from an internet-enabled machine. To configure internet connectivity on an instance of Brian Kellers Visual Studio ALM Virtual Machine, an external virtual switch with internet connectivity should be present in the Hyper-V manager. The ALM Virtual Machine should then have a network adapter associated with the external virtual switch added to it. Further information on creating a virtual network and adding a network adapter can be found here: http://aka.ms/configurevm
Synchronizing the Date and Time

Brian Kellers Visual Studio ALM Virtual Machine is configured with Time-synchronization disabled in order for the sprint backlogs and reports to work properly. This causes an SSL exception when connecting to the Perfecto mobile service: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. In order to fix this issue, the VM should be set to synchronize time with the host. Follow these steps to enable time-synchronization: 1. 2. On the Virtual Machine settings, select the Integration Services tab Check the time synchronization checkbox.
Once these steps have been completed successfully, the Perfecto Mobile TFS Add-in can be installed following the instructions detailed in the Local installation section.
Accessing the Perfecto Mobile Cloud Platform

This section will address the process and steps to access the Perfecto Mobile Cloud platform for a new or an existing Perfecto Mobile user. The Perfecto Mobile cloud platform is accessible via the URL https://demo.perfectomobile.com. Proper login credentials are required to work with Perfecto Mobile TFS add-in and gain access to the aforementioned URL. You register for a user account at the Perfecto Mobile at the https://demo.perfectomobile.com site. To use the Perfecto Mobile service, you have two options: As a new user As an existing user
http://aka.ms/almvms
Page 17 of 49
New User Access

New Users must register to gain access to the demo site. Access Registration by entering the URL specified above in a browser, and then navigate to the Click Here link as shown below.
Figure 5 - New User Registration link.
This link takes you to Perfecto MobileCloud registration page where you supply your registration information.
NOTE
Make note that the web page screen shots of the partners web site is subject to change from what we have shown you in this guidance.
Page 18 of 49
Figure 6 - Mobile Cloud Platform Registration.
Once the I Accept button has been clicked, the relevant login credentials will be sent to the supplied email address. User credentials are activated and can be used immediately.
Existing User Access

If you have an existing Perfecto Mobile account then login via the Perfecto Mobile URL as detailed above. After entering your credentials and clicking Sign In, the landing page will display.
Page 19 of 49
Hands-On Lab
If you are intending to follow the Hands-on Lab (HOL) step by step, use these times for each exercise as a guideline. If, however, you are intending to investigate each step in detail double the times as a bare minimum.
NOTE
This hands-on lab includes the following exercises: 1. 2. 3. 4. 5. Accessing the Perfecto Mobile Cloud and running an automated test Run an automated test in the Perfecto Mobile Cloud within Visual Studio Run an automated test in the Perfecto Mobile Cloud within Microsoft Test Manager Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on TFS Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Build Service on Team Foundation Service in the cloud Estimated time to complete this lab: 55 minutes.
Exercise 1 Accessing the Perfecto Mobile Cloud and running an automated test (10 min)
GOAL
1. 2.
NOTE
In this exercise you will use the perfecto mobile cloud, to run a predefined automated test against the Bank of America app on a specific mobile device. This automated script will run a functional test.
Persona Mapping
This exercise is focused on Christine as a tester.
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine5, you can log on as Julia Ilyiana with the standard password P2ssw0rd. Open Internet Explorer and browse to the URL https://demo.perfectomobile.com. Log on with your username and password.
If you dont have an account, please request your own demo account under: https://www.perfectomobile.com/trial/sign_up.
3.
NOTE
5
Click on Automation in the menu bar.

When an information window New in Automation! appears, click OK.
http://aka.ms/BrianKellerAlmVms
Page 20 of 49

4. Click File and Open to open an existing automation script:
Figure 7 - Switch to the automation area.
5. 6.
Select folder Public Scripts\Microsoft\ and open the script search_for_atm. The file is a script for test automation and contains different automated test steps:
Figure 8 - List of automated test steps.
7.
Click Running View to switch to the test steps execution status view.
Page 21 of 49

8. 9. Click on Run in the menu bar to start the automated test. Select the device Apple iPhone-4 for the field DUT, use the value iOS in the field OS to run the test on this specific device and click Run:
Figure 9 - Device selection popup window.
10. The test execution starts and the status for each test step :
Figure 10 - Running view with test script status list.
11. Click Watch in the menu bar while the test is running to switch to the Devices Live video view:
Page 22 of 49
Figure 11 - Devices Live video test execution view.
NOTE
The test script runs through the sign in process of the Mobile Banking app and finds the ATM location nearest to a specific zip code.
12. When the test script completes successfully a test report is available. Click Yes to review the report results.
Page 23 of 49

13. Click on General to check the test run result summary. There are information available about status, test duration, tested device, etc.:
Figure 12 - General view of the test run results.
14. Click on Execution Sequence to review the individual test steps and recorded screenshots. Scroll down to the last Text checkpoint to verify, that the expected text values Expected text (needle): sign in was found in the loaded app page and that the elapsed test step time didnt reach the set value for Timeout: 40 sec.
Page 24 of 49
Figure 13 - Execution sequence view for all test steps.
15. OPTIONAL: Click also on Errors to see if any errors occurred during test execution (should be empty in our case). 16. OPTIONAL: Click also on Statistics to get an overview about the passed vs. failed Checkpoints and Actions.
This exercise you ran an automated functional test against the mobile banking app provided by the company Bank of America. For testing you used the perfecto mobile cloud. You have learned how to open an existing test script and run that automation scripts against a selected mobile device. The generated test run report enabled you to verify the test run in detail.
REVIEW
Page 25 of 49
Exercise 2 Run an automated test in the Perfecto Mobile Cloud within Visual Studio (15 min)
In this exercise you will run a predefined automated test against the Bank of America app on a specific mobile device using Visual Studio. You will code an automated test that will access the perfecto mobile cloud and will run an automated test script as a functional test.
Persona Mapping
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine6, you can log on as Julia Ilyiana with the standard password P2ssw0rd. 1. 2. From within Visual Studio, create a new project via Ctrl+Shift+N. Select the Mobile Cloud Project template for Visual C# in the folder Installed -> Templates -> Visual C# -> Test and click OK:
GOAL
Figure 14 - Mobile Cloud Project template.
NOTE
If dont see the Mobile Cloud Project template, please install the perfectomobile-tfs-add-in-installer-1.0 extension for Visual Studio 2012.
3.
Open code file MobileCloudTest1.cs and overwrite the string values for mobileCloudServer, user and password in the public class MobileCloudTest1 with the following values:
// URL of the perfecto mobile cloud static String mobileCloudServer = "https://demo.perfectomobile.com/"; // user name, used for logon static String user = "<your username>"; // password, used for logon static String password = "<your password>";
Code 1 Parameters to initialize a connection to the perfecto mobile cloud.
Page 26 of 49

These parameters are necessary to connect successfully to the Perfecto Mobile Cloud during the test run.
NOTE
If you dont have an account, please request your own demo account under: https://www.perfectomobile.com/trial/sign_up.
Overwrite the string values scriptKey and DUT and add a parameter os in the test method ExecuteScript1():
// path to the automate test script in XML format String scriptKey = @"PUBLIC:Microsoft/search_for_atm.xml"; // Device ID of the emulated mobile device String DUT = "C1C30569F52259AD3E081816C30E54457DA55400"; // expected OS of the emulated mobile device String os = "iOS"; Dictionary<String, String> runTimeParameters = new Dictionary<String, String>(); // if needed, add additional runtime parameters below the DUT parameter runTimeParameters.Add("DUT", DUT); runTimeParameters.Add("os", os); // execute script with mobile cloud adapter adapter.Execute(scriptKey, TestContext, runTimeParameters, MobileCloudAdapter.REPORT_FORMAT_HTML);
Code 2 Coded start parameters for the test method ExecuteScript1.
NOTE
The value for the parameter DUT means the Device ID. You can find this Device ID in the perfecto mobile cloud portal in the list of devices as a separate column.
4. 5.
Build the solution with Ctrl + Shift + b and select TEST -> Windows -> Test Explorer in the menu to open the Test Explorer. Now you can see the test method ExecuteScript1 in the Test Explorer. To run the test method right click on ExecuteScript1 and select Run Selected Tests:
Figure 15 - Test Explorer: Run test method ExecuteScript1.
Page 27 of 49

6. OPTIONAL: While the selected test is running it is possible to view the script execution on the real mobile device in the Perfecto Mobile Cloud. To see the execution in real time open Internet Explorer and browse to the URL https://demo.perfectomobile.com/nexperience/dashboard.jsp?autoWatch=true.
Figure 16 See test execution on the real device in the cloud dashboard.
7.
After a while you will see the test result. The test method ExecuteScript1 passed. Click on the link Output to open the test result in a separate view:
Figure 17 - Test Explorer: Passed test method ExecuteScript1.
Page 28 of 49

8. Click on the link in the area Attachments PUBLIC_Microsoft_search_for_atm
Figure 18 - Test Output: Test method ExecuteScript1.
and you will see a html based test execution report:
Figure 19 - Html-based test execution report summary.
9.
Scroll down to see the execution time for each test step in the area Execution Sequence:
Figure 20 - Html-based test execution report with execution sequences.
Page 29 of 49

NOTE
If the test fails, the html based test execution report shows which execution sequences failed.
10. Scroll down to see the device information in the area Devices and click on the link Video ():
Figure 21 - Device information.
11. A video starts and shows the recorded test execution:
Figure 22 - Recorded test sequence video window.
In this exercise you ran an automated functional test within Visual Studio against the mobile banking app provided by the company Bank of America. For testing you used the perfecto mobile cloud. You have learned how to create a new Visual Studio Mobile Cloud project. You modified the provided test code to execute an existing test script and run that automation script against a selected mobile device. The generated html based test sequence report enabled you to verify the test run in detail. The recorded test execution video was helpful to visually verify the test execution on the mobile device.
REVIEW
Page 30 of 49
Exercise 3 Run an automated test in the Perfecto Mobile Cloud within Microsoft Test Manager (10 min)
The Perfecto Mobile Cloud can also be accessed from within a Microsoft Test Manager (MTM) test case. In this exercise, you will create a test case and enable automation using the unit test you created in exercise 2 so that you can see the end to end ALM integration for mobile device testing. Microsoft Test Manager requires integration with TFS where unit testing does not. The test project and unit test file created in Exercise 2 will need to be checked into your TFS repository so that the test plan you create can access the unit test.
Persona Mapping
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine7, you can log on as Julia Ilyiana with the standard password P2ssw0rd. 1. 2. From within Microsoft Test Manager, under the Plan tab, create a new test case. Enter the title of the Test Case. There is no need to add any Test Steps.
NOTE
GOAL
Figure 23 - Create a new test case.
NOTE
The automation status of the Test Case is set to Not Automated. You will need to use Visual Studio to add the associated automation as listed in the following guided steps.
3. 4.
After saving your test case, open Visual Studio and query the test case previously created. Navigate to the Team menu item in the menu bar within Visual Studio, click on Go to Work Item and provide the id number for the test case that was created in Microsoft Test Manager.
Page 31 of 49

5. Once the test case has opened, navigate to the ASSOCIATED AUTOMATION tab as shown below:
Figure 24 - Test Case associated automation.
6.
Click on the ellipse button. The test name previously created should appear. Highlight the test name and click OK as shown below.
Figure 25 MTM: Associate the test case with the test script.
Page 32 of 49

7. After saving the test case, the Unit Test will now be associated with the Test Case.
Figure 26 MTM: Test case details.
You will now need an Environment to run the test case with automation. If you do not have an Environment, you will need to create one. If you do not have an Environment configured, the following steps will guide you through the process of creating a new environment to run the Test Case.
NOTE
8.
Within Microsoft Test Manager, navigate to the Lab Center, select the Lab tab, and click New:
Figure 27 MTM: Create a new environment.
Page 33 of 49

9. Select Standard Environment and provide a suitable name. Then click Next:
Figure 28 MTM: Set environment name.
10. In the Machines tab, select Add Machine. 11. Provide the name of the computer from where the tests will run. 12. Follow the steps and click Verify:
Figure 29 MTM: Add machine
13. Once verification is complete, a new Environment is created and initialized in a running state. 14. The Environment can now be used by associating it in the Test Settings tab in the Lab Center. 15. Within Lab Center, select the Test Settings tab.
Page 34 of 49

16. Click New, provide the Name and select the tests type for Automated as shown below:
Figure 30 - MTM: Configure Environment with Test Settings.
17. Click Next to navigate to the Roles tab and select the Environment created previously. 18. Click Finish to create a new Test Setting. The completed Test Setting is shown below:
Figure 31 MTM: Completed Test Settings.
19. Next, navigate back to the Test Center and click on the properties in the Plan tab. 20. Select the Test Settings and Test Environment created previously.
Figure 32 MTM: Plan properties.
Page 35 of 49

21. Now click on the Modify link under Builds: Build in use section of the properties, click the Assign to plan button to associate the build triggered in the Visual Studio steps previously. The completed properties tab should look as follows:
Figure 33 MTM: Completed Properties tab.
22. Navigate to the Test Lab in Test Center and under Run select Run with options. Pick the selection that was created in the properties section and click Run as shown below.
Figure 34 MTM: Select run options.
Page 36 of 49

23. OPTIONAL: While the selected test case is running it is possible to view the script execution on the real mobile device in the Perfecto Mobile Cloud. To see the execution in real time open Internet Explorer and browse to the URL https://demo.perfectomobile.com/nexperience/dashboard.jsp?autoWatch=true.
Figure 35 See test execution on the real device in the cloud dashboard.
Allow enough time for the test to pass depending on your network connection. A successful test run is shown below.
Figure 36 - MTM: Successful test run.
The Perfecto Mobile Cloud for TFS add-in will attach the test results to the test case run. Once again, the results are an html file showing the sequences as they were executed within the Perfecto Mobile Cloud. This is shown below:
Page 37 of 49
Figure 37 MTM: Html-based test execution report with execution sequences
In this exercise you created a test case and enabled automation of the functional test within Visual Studio against the mobile banking app provided by the company Bank of America. For testing you used the perfecto mobile cloud. You have learned how to integrate a test created in a Visual Studio Mobile Cloud project with Microsoft Test Manager. The generated html based test sequence report enabled you to verify the test run in detail.
Exercise 4 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on TFS (10 min)
The Perfecto Mobile Cloud can also be accessed from within a Team Build. In this exercise, you will create a new build definition and enable testing within Continuous Integration using the unit test you created in exercise 2 so that you can see the end to end ALM integration for mobile device testing during the build process. Continuous Integration requires integration with TFS. The test project and unit test file created in Exercise 2 will need to be checked into your TFS repository so that the Team Build can access the unit test. Follow the steps from the Exercise 2 on the Hands-on Lab first to create a Mobile Cloud Project in Visual Studio and start with this Exercise.
Persona Mapping
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine8, you can log on as Julia Ilyiana with the standard password P2ssw0rd. Before running Perfecto Mobile test cases as part of a Continuous Integration build with Team Foundation Server, the solution from Exercise 2 must be checked-in to TFS Source Control by following these steps: 1.
8
NOTE
GOAL
REVIEW
From Solution Explorer, Right-click on the solution.
Page 38 of 49

2. Select Add Solution to Source Control:
Figure 38 Adding the Solution to Source Control.
3.
On the Add Solution to Source Control dialog, select a Source Control Path and click OK:
Figure 39 Selecting the Source Control path.
4.
Right-click on the solution again and select Check In.
Page 39 of 49

5. In the Pending Changes tab of Team Explorer, enter comments and click on Check In:
Figure 40 Checking in the solution.
6.
Once the solution is checked in, create a new build definition by going to the Builds tab in Team Explorer and clicking on New Build Definition:
Figure 41 Creating a new Build Definition.
Provide a Name for the new Build Definition and select all the defaults. On the Process tab, verify that your solution is set as the value for Projects to Build. Expand the Automated Tests section in the build definition, then Test Source and make sure that the Test Source Spec argument includes the name of your test assembly. By default, this value is **\*test*.dll. If you selected the default name when creating the Perfecto solution, your assembly name will be Mobile Coud Project1, which wo uld not be picked up by the build process. You should update this value to **\**.dll.
Page 40 of 49
Figure 42 Setting Build Definition Arguments.
You may now queue the build and your Perfecto Mobile test case will run during the build.
Figure 43 Build Results, including completed test case.
Exercise 5 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on Team Foundation Service in the cloud (10 min)
The Perfecto Mobile Cloud can also be accessed from within a Build based on Team Foundation Service in the cloud. In this exercise, you will create a new build definition and enable testing within Continuous Integration using the unit test you created in exercise 2 so that you can see the end to end ALM integration for mobile device testing during the build process. The difference between this Exercise 5 and the Exercise 4 before is, that Continuous Integration will now run in the cloud. Continuous Integration requires integration with TFS. The test project and unit test file created in Exercise 2 will need to be checked into your TFS repository so that the Team Build can access the unit test. Follow the steps from the Exercise 2 on the Hands-on Lab first to create a Mobile Cloud Project in Visual Studio and start with this Exercise.
Persona Mapping
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine9, you can log on as Julia Ilyiana with the standard password P2ssw0rd. When you install Perfecto Mobile on a development machine, a set of assemblies get copied to C:\Program Files (x86)\PerfectoMobile\Visual Studio. Since Perfecto Mobile cannot be installed on TFS Service, those
NOTE
GOAL
Page 41 of 49

assemblies should be included as part of our solution so that they get included with the build and are available when the test runs during the build. The assemblies needed are: PerfectoMobile.MobileCloudHttpClient.dll and PerfectoMobile.TFS.Adapter.dll.
Figure 44 Adding Perfecto Mobile assemblies to a solution.
Before running Perfecto Mobile test cases as part of a Continuous Integration build, the solution must be checked-in to TFS Source Control by following these steps: 1. From Solution Explorer, Right-click on the solution and select Add Solution to Source Control:
Figure 45 Adding the Solution to Source Control.
Page 42 of 49

2. On the Add Solution to Source Control dialog, select a Source Control Path and click OK:
Figure 46 Selecting the Source Control path.
3. 4.
Right-click on the solution again and select Check In. In the Pending Changes tab of Team Explorer, enter comments and click on Check In:
Figure 47 Checking in the solution.
5.
Once the solution is checked in, create a new build definition by going to the Builds tab in Team Explorer and clicking on New Build Definition:
Page 43 of 49
Figure 48 Creating a new Build Definition
Provide a Name for the new Build Definition and select all the defaults. On the Process tab, verify that your solution is set as the value for Projects to Build. Expand the Automated Tests section in the build definition, then Test Source and make sure that the Test Source Spec argument includes the name of your test assembly. By default, this value is **\*test*.dll. If you selected the default name when creating the Perfecto solution, your assembly name will be Mobile Coud Project1, which would not be picked up by the build process. You should update this value to **\**.dll.
Figure 49 Setting Build Definition Arguments.
You may now queue the build and your Perfecto Mobile test case will run during the build.
Figure 50 Build Results, including completed test case.
Page 44 of 49
Conclusion
We have shown you how easy it is to integrate the Perfecto Mobile Cloud for TFS with Visual Studio. Testing of your mobile application running on real devices in the cloud is possible as strictly unit tests or as automation tests associated to a test case in Microsoft Test Manager. These integration points allow these tests to be run as part of a build definition for daily, manual, or continuous integration builds. This integration provides the test status reporting features, like status of tests (pass/fail) along with the failure reason. This document does not cover the setup details required to develop for iOS, Android, or Windows Phone. Team Explorer Everywhere 10 is required to enable developing and building Android and iOS applications. For an overview of the lifecycle with Mobile Cloud see the overview poster for this ALM Ranger project.
10
http://msdn.microsoft.com/en-us/library/gg413285.aspx
Page 45 of 49
Appendix
Automation Tool Comparison
Feature Supported Mobile OS SeeTest from Experitest Android iOS Blackberry WindowsPhone8 All emulators are supported See Test does not provide devices. Connect all mobile devices to one central hub/server that resides in your organization to form private cloud Enable remote access from anywhere in the organization to any device Object identification by: Native ID, HTML5-DOM, Image Recognition and Text Recognition Browsers testing Reboot device Awake device from sleep mode Operate device power button Operate device Back, Call/Hang Call, Home, Menu buttons Shell commands for internal device counters (e.g. CPU, memory) Landscape model Object identification by: id, class, text, accessibility Label, accessibility Identifier. Simple recorder Object spy for easy identification of object native properties Easy instrumentation with no code change PerfectoMobile Android iOS Blackberry
Remote Access
Provides devices which are accessed over cloud.
Test Coverage
Supports Visual Objects, Native Objects, DOM Objects Get Device Info, state, log Open Device, Reboot device, rotate device Keyboard display Cloud call, SMS, email
Native Object Identification
Supports Native Objects, DOM Objects Object Spy
Page 46 of 49

Web & Hybrid Object Identification Object identification by: id, name, class, text, css. JavaScript execution. Support for native browser Operations like click, send text, select, touch and others. jQuery support. Object Spy for easy identification of object web-DOM / hybrid properties Easy instrumentation with no code change OCR engine with support for all languages Text input Standard icons/images (e.g. buttons) Proprietary icons/images Auto-sense capabilities for oneclick image extraction Sensitivity calibration to changes in image (e.g. size, color, background, transparency) Image identification on changing backgrounds (e.g. transparent, gradient, moving elements) Supports Native Objects, Web Objects Object Spy Install/Uninstall application
Text Object Identification
Supports OCR Text input
Image/Icon Object Identification
Allows simple and intuitive method for pressing virtual locations on touch handsets based on text or image analysis. The system automatically finds those elements, even if their location changes, without any change to the script itself
Gesture Support
Multitouch Swipe Drag & Drop Zoom Scroll inc. conditional scrolling
Swipe Touch Trackball
Keyboard Support
Device keyboard: All configurations (alphabet, punctuation and numbers, and punctuation and symbols) All languages PC keyboard special characters and capitalization
Regardless of the method implemented by the handset (physical or virtual keyboard, different languages, capitalization), simple type text function hides the complexity of any string typing
Page 47 of 49

Scripting Capabilities Recorder Script debugger Script Multiplier: Script Once, Run on Multiple Devices Can convert SeeTest to different programming languages e.g. C#,Java etc. and hence extensibility is high Plug-in to QTP MS Test Test Complete Perl Python JUnit3, JUnit4, TestNG Full functionality supported: Scripting, editing and reporting (inc. screenshots) capabilities directly from within test tools Full and seamless integration to QC,TFS,JUnit, NUnit, PyUnit, TestNG, Ant Capture the screen shots and gives summary reports Recorder Script Multiplier: Script Once, Run on Multiple Devices Perfecto Script cannot be converted to any programming languages
Plug-in to Existing Testing Frameworks
Plug in to QTP MS Test
Continuous integration support Comprehensive Reporting
Full and seamless integration to QC,TFS etc. Capture screenshots and record videos Share test reports with other colleagues and partners, Share devices with others to demo new apps/features, reproduce a bug.
Table 5 Leading Automation tools comparison
FAQ
Which type of Mobile applications need to be tested on different mobile networks like 2G/3G/4G LTE Conditions? For specialized applications where on location testing for specific cellular networks is to be considered like Free usage allowance or two way SMS Why to test such applications on different mobile networks? Mobile networks have limited bandwidth and high latency (compared to Wi-Fi and broadband) which effects the application performance by increasing the page load time on mobile device resulting in testing an application in real world conditions How to test an application on different mobile networks? Network simulation tools helps in reproducing different network conditions by limiting bandwidth and simulating latency as a result help you in testing whether all mobile users will be getting best user experience in acceptable response times on mobile devices
Page 48 of 49

Device anywhere and perfecto are also good solutions if it is required to test an application on a particular networks like US Network, Canada network Check list for network testing of mobile applications:- Test behavior of application when there is no network or network problem is there. Test the application on all supported network connections. Test application does not hang whenever there is a change in the signal strength. For a device it should switch to mobile network when there is no Wi-Fi and back to Wi-Fi when there is both WI-FI and mobile network Test the robustness of the application in low bandwidth/high latency scenario. Is the application giving the correct error whenever is some network related issue rather than any exception or false error.
Page 49 of 49

Testing Mobile Applications Using Perfecto Mobile MobileCloud V2

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Testing Mobile Applications Using Perfecto Mobile MobileCloud V2

Încărcat de

Drepturi de autor:

Formate disponibile

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

2013 Microsoft Corporation.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Visual Studio ALM Rangers

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Table 1 Possible test execution error scenario.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Device testing approach for IT Applications

Figure 1 - PC verses Devices Spread across verticals

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Challenges in Device Testing

Device Testing Lifecycle for IT Applications (DTL-IT)

Figure 2 - Device testing life cycle for it applications

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Device Configuration Matrix

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Real /Virtual Mobile Testing Decision Table

Table 3 Mobile testing decision table

Device Testing Cost Pyramid

Figure 4 - Device testing Cost Pyramid

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Test Automation Tools for Device testing

Automation is Not Recommended for the following areas

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Useful tools for developers to catch bugs early

W3C mobileOK Checker

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Table 4 Device testing tools

Device Usability Testing

Combinatorial Testing to generate effective number of scenarios

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Security Testing on Devices

Native App Best Practices

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Installing the Perfecto Mobile Team Foundation Server (TFS) addin

Performing an Local Installation

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Installing to a Hyper-V Virtual Machine

Installing to the Brian Keller Virtual Machine

Enabling Internet Connectivity

Synchronizing the Date and Time

Accessing the Perfecto Mobile Cloud Platform

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

New User Access

Figure 5 - New User Registration link.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Figure 6 - Mobile Cloud Platform Registration.

Existing User Access

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Click on Automation in the menu bar.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Figure 7 - Switch to the automation area.

Figure 8 - List of automated test steps.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Figure 9 - Device selection popup window.

Figure 10 - Running view with test script status list.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Figure 11 - Devices Live video test execution view.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Figure 12 - General view of the test run results.

ALM Rangers - Testing mobile applications with Perfecto MobileCloud