Proactive Application Performance Testing to Prevent Outages

Addressing the commonly faced challenges by app developers in providing customers with a smooth and reliable user experience.

The apps market is growing in leaps and bounds and new features are being introduced almost every day to stay ahead of competition. Just to put it in perspective, the Apple App store has a base of nearly two million and Google play, over 2.2 million apps. Gartner predicts that an income of $77 billion will be generated in 2017 from over 268 billion mobile downloads. Compare this to $35 billion revenues in 2014, $45 billion in 2015, and $58 billion in 2016. The primary target is the age group falling between 18 and 24 years followed by 25 to 34 years.

Retaining customer interest and loyalty will be one of the greatest challenges for app developers as customers are spoilt for choice. Performance testing will become critical and a game changer determining who leads the pack.

The Challenges to Successful Implementation

App developers face several challenges that can hurt the performance of the app at various stages:

Three key mistakes that can turn away customers from retail / eCommerce mobile apps and website performance:

• Not optimizing the images
• Not compressing the files
• Not using the browser cache

Top 3 mobile app performance issues encountered in Cloud based load testing

• Often sites use different JavaScript libraries for Android and Apple iOS, which can get ‘out of sync’. The result is Java Script on a page that is no longer compatible with all page-types.
• Failure to incorporate the small rendering/DOM quirks between browsers on Android and iOS
• Incompatibility between third party library interactions

Top five capabilities that need to be effectively addressed to avoid performance challenges that impact the business or revenue model of the mobile application

• Ability to segment the delay in application response times as server, network, and application delay
• Ability to translate application performance metrics into business metrics such as page views, conversions, employee productivity, and customer satisfaction
• Develop capabilities for correlating application performance metrics with appropriate business metrics (customer satisfaction, revenue growth, brand reputation, etc.)
• Identify parts of the enterprise infrastructure that are causing the deterioration of business services
• Ability to measure application performance from a business user’s perspective

Understanding and assessing the business impact of the performance issues pertaining to mobile applications

• One additional second of delay beyond defined baselines for performance of Mobile / Web applications can cause loss of business revenues. Studies show end users are willing to wait typically between 4 and 6 seconds for a page to open before they abandon a session and move on to the next app / website in their search.
• Brand perception drops among existing customers while also driving away new customers, thereby increasing the churn out ratio.

A comprehensive, planned performance testing process can help developers overcome all of these challenges before the release, thus enhancing user experience. Indium’s jumpstart kits and iAccelerate automation frameworks further reduces testing time, meeting the needs of app developers to deliver on time and with quality.

Indium’s performance testing process is aimed at increasing revenues, expanding user base and growing the business.

Top Performance Testing Tools

What happens when millions of users with their unique credentials try to login at a time?
How will you forecast and prevent performance outages?

The best way this situation can be handled is through – virtual user (VU). For all our tests, the VU behaves just the way a real user would.
For the creation of as many VUs, as you would require and simulate real-time conditions, performance testing tools are employed. Not only that, Performance testing also tests for the peak load usage, breakdown point, long-term usage, etc.

To enable all with limited resources fast and to obtain reliable results, tools are often used for this process. There are a variety of tools available in the market- licensed, free wares and open sourced.

Check out our list which covers top performance testing tools (both open-source as well as licensed).

Open Source Performance Testing Tools

1.Apache JMeter

Open source software and 100% pure Java application which is designed to load test functional behavior and measure performance.

Features

• Full-featured Test IDE that allows fast Test Plan recording
• Dynamic H TML reporting
• Full multithreading framework allows concurrent sampling
• Complete portability
• Highly Extensible core

2.Grinder

The Grinder is a popular Java-based framework for load testing. Jython (the special Java implementation of Python) is the default script language. TestScripts also could be written in Clojure.

The main parts of The Grinder:

• The Grinder Console is the basic interactive development environment (IDE) for test suites developing and editing. It monitors results in real time and controls various agents of The Grinder.
• The Grinder Agents are the special headless load generators. They have a number of Workers to create the load. 

Key features of Grinder

• It is a cross-platform tool. It could run on any platform and a Java Virtual Machine can be set up immediately.
• Multiple protocols support.
• Rapid and powerful script development
• Simple, sliding performance graphs for all transactions in one tab.
• The users have an ability to write the own plug-ins to a documented interface
• Flexible parameterization.
• The user has an ability to load Java API as a test from HTTP web servers, SOAP and REST services.
• Flexibility for loading and executing third-party libraries.

3.Gatling

Gatling is the powerful Scala-based “firearm” for load testing with two executables:

• one for test case recording
• other for test case execution

The main features of Gatling are listed below

• It works with almost all operating system and any browser.
• Gatling supports colorful and informative graphical reports containing valuable information for analysis.
• It can execute its test cases in different test clouds.
• Gatling could be easily integrated with Jenkins through the Jenkins-plugin and runs the tests through Gradle and Maven with the help of Maven plugin and Gradle plugin.
• Gatling uses only a small amount of memory.
• It reduces the debugging phase time.
• Errors and drawbacks are detected in the early part of the development cycle.

4.Locust

Locust is a code-driven, distributed load testing suite built in Python. It is frequently used for load testing websites and indicating how many concurrent users a system can handle.

According to it, the swarm of locusts will attack the website during the test. This tool can define the behavior of each locust that is the user. The swarming process is monitored from a web UI in real-time and helps to identify the weak points of the code.

Locust’s key features

• Any system or platform can be tested with this tool.
• It is the cross-platform and easily extendable tool.
• Hundreds of thousands of users’ behavior can be simulated with Locust.
• Locust has reusable codes.
• It has the high level of customization. There is an ability to combine several requests as one scenario by grouping requests with specific custom points.

5.Tsung

Tsung is a distributed load testing tool which is available free of charge as an open source software product. It is protocol-independent and can be used to stress Jabber/XMPP, HTTP, SOAP, LDAP and PostgreSQL servers. The tool can simulate very large numbers of users per server, making it ideal for analyzing and testing the performance of large-scale applications, such as instant messaging solutions.

High performance and ease of use are the main features of this tool. Tsung can simulate up to 50,000 simultaneous users per physical computer, whereas traditional injection tools can usually only simulate up to 1,000 users. Based on XML configuration, Tsung removes the need for developers to write complex scripts. The tool handles tasks such as managing cookies automatically, so developers can focus on designing the scenarios not managing lines of code.

Originally known as IDX-Tsunami, Tsung was first developed in 2001 by a company called IdealX. Its current name stands for Tsunami Next Generation. Developed in the Erlang programming language, the tool is now an open source solution that is actively developed and supported by ProcessOne.

Commercial Tools for Performance Testing

1.LoadRunner

This is a HP product which can be used as a performance testing tool.  This can be bought as a HP product from its HP software division.  Also, it is very much useful in understanding and determining the performance and outcome of the system when there is an actual load. One of the key attractive features of this testing tool is that it can create and handle thousands of users at the same time. This tool enables to gather all the required information with respect to the performance and also based on the infrastructure.  The LoadRunner comprises of different tools; namely, Virtual User Generator, Controller, Load Generator and Analysis.

LoadRunner System Requirements: Microsoft Windows and Linux are the favorable OS for this measuring tool.

2.Neoload

NeoLoad is a load and stress testing tool to measure the performance of web and mobile applications. NeoLoad provides pragmatic solutions to developers to optimize the performance before the application goes into production.

Using the script less GUI and a host of automation features, NeoLoad lets you design tests 5-10x faster and maintain lesser scripts which are as little as 10% of the original design time while helping to automate testing with Continuous Integration system with the agile feature.

3.Rational Performance Tester

The Rational performance tester is an automated performance testing tool which can be used for a web application or a server-based application where there is a process of input and output is involved. This tool creates a demo of the original transaction process between the user and the web service. By the end of it, all the statistical information is gathered and they are analyzed to increase the efficiency. Any leakage in the website or the server can be identified and rectified immediately with the help of this tool. This tool can be the best option in building an effective and error free cloud computing service. This Rational Performance tester was developed by IBM (Rational software division). They have come up with many versions of this automated testing tool.

Rational Performance Tester System Requirement: Microsoft Windows and Linux AIX good enough for this performance testing tool.

4.WebLOAD

WebLoad is a Load and performance testing tool for web applications. WebLOAD allows to perform load and stress testing on any internet application using Ajax, Adobe Flex, .NET, Oracle Forms, HTML5 and many more technologies. The load can be generated load from the cloud and on-premises machines. WebLOAD’s benefits are its ease of use with features like DOM-based recording/playback, automatic correlation, and JavaScript scripting language. The tool supports large-scale performance testing with heavy user load and complex scenarios and provides clear analytics on the functionality and performance of the web application. Another highlight of the tool is its flexible licensing mechanism and pricing.

System Requirements: Windows, Linux

5.WAPT

Performance testing tool for websites and intranet applications: WAPT refers to the Web Application Performance tool.  These are scales or analyzing tools for measuring the performance and output of any web application or web related interfaces. These tools help us to measure the performance of any web services, web applications or for any other web interfaces. With this tool, you have the advantage of testing the web application performance in the various different environment and different load conditions. WAPT provides detailed information about the virtual users and its output to its users during the load testing. This is considered to be the best cost-effective tool for analyzing the performance of the web services. The WAPT tools can tests the web application on its compatibility with the browser and operating system. It is also used for testing the compatibility with the windows application in certain cases.

WAPT System Requirement: Windows OS is required for this testing tool

• Microsoft Testing Suite (VSTS)

6.SmartMeter.io

SmartMeter.io is a performance and load testing tool using a familiar JMeter interface, yet adding enterprise-level features.

SmartMeter.io makes it super simple to create a test scenario thanks to its state of the art test recorder. It lets you create test scenarios just by clicking in an embedded browser. There’s no proxy setup or browser plugin.

It also automatically generates test reports with all details about the test and its results containing graphs, statistical overview, evaluated acceptance criteria, trend analysis, and a graph comparison tool.

Among other features, it is strong in distributed testing, CI integration, and offers unparalleled performance testing support for Vaadin apps. Compatible with Windows, Linux, and macOS.

7.LoadUI

Create sophisticated load tests with just a few clicks.

With LoadUI NG Pro it is easy to model real-world loads on the API. LoadUI supports REST, SOAP, JMS, MQTT and many other API formats. The tool has an easy-to-use graphical interface which makes it simple for new users to setup load scenarios, while also providing advanced scripting features for those with more experience.

This tool allows to analyzing results than configuring and building tests by hand. The tool can also be used to report server performance data. This tool allows to setup local or globally distributed load agents for the load testing scenarios. LoadUI NG Pro allows easy reuse of existing functional test cases from SoapUI and SoapUI NG Pro.

System Requirements: Windows, Linux, and Mac OS

8.LoadComplete

SmartBear LoadComplete is a desktop tool for load, stress, scalability testing of websites and web applications. Designed from the ground-up to help you focus on the tasks at hand, it does not require advanced programming skills. It works with both traditional HTML websites and with Rich Internet Applications using modern web technologies like AJAX, ASP.NET, Java, Flash, Flex, and Silverlight. The ‘record and playback’ and ‘visual programming’ features enable you to create powerful and flexible first load test within a few seconds.

LoadComplete allows you to generate load from virtual machines, on-premise computers, or even the cloud. Customized reports in LoadComplete help you get end-to-end visibility and find problems at both client and server side.

Top Cloud-Based Performance Testing Tools

1.StromRunner Load

Simple and scalable load testing tool used to design & create mobile and web load test within a short duration of time for easy cloud-based testing without the need to schedule, deploy, and manage load generators. It creates test scripts in the browser using HPE’s patented TruClient technology, an innovative scripting tool based on node.js. StormRunner supports LoadRunner Web, Performance Center, Web Services, Flex, Mobile and TruClient protocols or it can simply upload from REST call, HAR file, CSV or import JMeter scripts.

The tool detects problems fast. It emulates and tests real-world network conditions with Network Virtualization to get accurate and repeatable results. It provides a comprehensive root cause, trending, and regression analysis as well as the intelligent and predictive analytics for a holistic client, network, and server-side breakdown report. StormRunner integrates with Jenkins, Gatling, Dynatrace, New Relic, Docker, AWS Code Pipeline and much more.

With this tool, we can Scale from 1 tester to 1,000,000 or more geographically distributed web and mobile users, add or remove them during runs.

2.BlazeMeter

With BlazeMeter, developers/testers can easily create tests (or harness existing open source scripts) in the favorite editor using straightforward YAML or JSON syntax. With this tool, one can create and edit performance tests alongside code without switching context or having to open other tools. With this keep the tests in the preferred version control repository to track changes and revert when necessary.

3.Loadstorm

Loadstorm is a cloud load testing tool for web applications. Loadstorm is the cheapest available performance and load testing tool. It has the option of creating required test plans, testing criteria and testing scenario. It can generate up to 50000 concurrent users by generating traffic to your website and then carry out the testing. This tool can bring an end to all the expensive performance testing tools. The cloud infrastructure is used in this tool, which enables you to send a huge amount of requests per second. There are thousands of servers available around the world for this software. They are proudly known as the lowest cloud load testing tool. There is no need of any scripting knowledge for using this tool. You will be provided with many graphs and reports which measure the performance in various metrics such as error rates, average response time and a number of users. This tool is available for free, but the premium account comes with added features.

System Requirement:  Windows OS.

4.CloudTest

SOASTA CloudTest is a performance testing tool for web, mobile apps and APIs. The users or the developers can use the cloud platform as their virtual testing lab. The developers can carry out the performance or load testing in the cloud platform in a cost-effective way through this tool. It has a capacity to enable user load for the website at the same time. It also increases the traffic of the website to know the actual performance under stress and heavy load.

CloudTest is not an open source tool; the price differs according to the number of load injector machines required per hour. The trial version with the power of 100 concurrent users is available for free.

Application Testing Vs Game Testing – Tug of War!!!

What Is Different About Software Testing?

Unlike game testing, software testing greatly utilizes automation scripts, and it is the matter of testing a software application, not games. These scripts can be created before or after producing written test cases, as either can be used to supplement the testing of the other similar application or program. A number of tools and frameworks have been developed to make automation quicker, as setting up automation scripts can initially take a bit of your time.

As software testing makes greater use of test cases and test scripts, this generally means that the job of a software tester is more technical than a games tester.

The field of software testing is usually seen as a more skilled profession than game testing, even though the skills required to be a games tester are exactly the same as those needed as a manual software tester.

“For software testing, you usually need a degree in computer science, software engineering, math, or anything technical, really.” 

“For games testing, a degree may be useful and a love of games, attention to detail, and a strong work ethic can usually get you into the field.”

Differences by Testing Platforms

Game Testing	Software Testing
PC, Consoles, Web/Cloud (online games likes MMOs, HTML5 and Facebook) Mobile	PC, Web, Mobile & Cloud

What Makes Game Testing Unique?

Below are the few testing types to the list of game testing checklist, which makes game testing unique.

• Fun Factor Testing
• Balance Testing
• Realism Testing
• Game Level Testing
• Game Play Testing
• Multi-player Testing
• Modification API Testing
• Game Network Testing
• Al Testing

A game tester should have the same knowledge as of software tester, but with a special focus on what makes

Game Testing is not an easy task. It requires more dedication and concentration. Now a day’s business requires for gaming consoles and game applications that are perfect from all viewpoints. For creating a good image and better response that are required to deliver games that are free from bugs and capable to run for across devices and platforms.

In games, the app is designed to give the user the least amount of access to the actions of the program as possible.

Game designers give the players with no choice but to take time to explore and discover what the game does. They want to preserve the mystery of the game world, the pursuit, the fantasy. The harder it is, and the more time it takes for the user to obtain the ‘desired results’. “Winning”.

A game is an enormously complex database or series of databases, within a shell that is built to obscure its purpose from the user.

Adding a feature of randomizer, the game design is unpredictable. Events don’t function the same way twice.

As a game tester, the interaction is only with an intentionally limiting interface that is constantly changing the way of its functions, as well as the way it looks.

“It is almost impossible to know if what is in the
screen is what is supposed to be seen”.

A game tester should be able to keep track of enormous amounts of dynamic details.

In a game’s database, it is not necessary that the same series of actions do not cause the same reaction. This makes recreating bugs extremely difficult.

Games push the technology innovation envelope. They tend to drive the hardware market, while other software conforms to it.

Gamers are often technology wizards, and are looking for the next level of performance, speed, and visual quality in their game experience. Games are always developed with an eye toward the newest, fastest, latest hardware, so game testers must keep ahead of the trends.

Testers must be educated and certified in all new hardware and have it all available in the testing environment. This technological knowledge is critical in this industry because the game developer will most often take the blame if it the game doesn’t function with the new hardware, and that failure can mean game death.

Games are more system demanding than most other software, so installation, setup, and compatibility testing is critical. Game function will be affected by video, sound, memory, and other issues more than any other type of software.

In contrast to that, the latest versions of today’s business software are released to be compatible with the standard issue in system configuration.

Conclusion

There are many similar and unique aspects between software testing and game testing but all types of testing require experts to perform it well and in a structured way.

Top iOS Testing Frameworks

Did you know that the total number of iOS devices sold in the
market is 250 Million+?
11% of the iOS users remove the app after crashes
45% of iOS users would remove the paid apps after crashes
69% of iOS users would will stop playing and remove the games after first crash
6-7 iOS devices are sold every second

With the explosion of iOS devices year on year, the verge of iOS success is increasing higher than ever before. More start-ups and SMEs don’t find it financially sustainable to acquire every piece of iOS devices with different OS versions and hardware specs. On the other hand, testing apps or games manually is not anymore an ideal option for QA process due to low efficiency and scalability issue.

Though iOS is still a more closed operating system compared to Android, many open source test automation frameworks and tools can be utilized to create robust automated tests. Along with the help of a cloud-based testing solution, it enables developers/testers to verify app quality more efficient in a scalable way and makes iOS testing tasks easier.

Today we’d like to go through 5 widely used iOS testing frameworks nowadays with simple code samples to give you a basic understanding of getting started with your iOS testing. If you are also interested in Android test automation frameworks, do not forget to check out Top 5 Android Testing Frameworks, where code samples are provided as well.

APPIUM

Appium is popular due to its flexibility and usability on both Android and iOS, and it works on native, hybrid and web applications. For iOS testing, it uses JSONWireProtocol to engage with iOS applications using Selenium WebDriver. Thanks for that, Appium does support mobile web testing very well and its use cases are very similar as if Selenium would be used for web testing.

APPIUM CODE SAMPLE

driver.findElement(By.id(“com.example.app:id/radio0”)).click();
driver.findElement(By.id(“com.example.app:id/radio1”)).click();
driver.findElement(By.id(“com.example.app:id/radio2”)).click();
driver.findElement(By.id(“com.example.app:id/editText1”)).click();
driver.findElement(By.id(“com.example.app:id/editText1”)).sendKeys(“Simple Test”);
driver.findElement(By.name(“Answer”)).click();
// or alternatively with
driver.findElement(By.id(“com.example.app:id/button1”)).click();

XCTEST / KIF

XCTest is tightly coupled with Xcode but is still usable with both real iOS devices and simulators. XCTest allows developers to write tests for components at any level and also provides the framework for UI testing capabilities. XCTest tests are grouped into subclasses of XCTestCase. Writing any tests with XCTest should be trivial to iOS developers as XCTest is fully compatible with both Objective-C and Swift.

KIF (Keep It Functional) is an iOS integration test framework that is closely related to and uses XCTest test targets. KIF tests can be executed directly in XCTestCase or any subclass. KIF allows for easy automation of iOS applications by leveraging the accessibility attributes that the OS system makes available for those with visual disabilities.

CODE SAMPLE WITH OBJECTIVE-C

– (void)testClicksOnRadioButtons {
[tester tapViewWithAccessibilityLabel:@”Radio1”];
[tester tapViewWithAccessibilityLabel:@”Radio2”];
[tester tapViewWithAccessibilityLabel:@”Radio3”];
[tester enterText:@”Simple Test”
intoViewWithAccessibilityLabel:@”editText1”];
[tester tapViewWithAccessibilityLabel:@”Answer”];
}

CODE SAMPLE WITH SWIFT

testClicksOnRadioButtons() {
let app = XCUIApplication()
app.radiobutton[0].tap()
app.radiobutton[1].tap()
app.radiobutton[2].tap()

app.staticTexts[“Simple Test”]
app.button[0].tap()
}

Learn and get started with KIF by reading our blog about this: How to get started with KIF?

CALABASH

Calabash is another great cross-platform framework that works perfectly with Android and iOS apps. One of the major differences to other frameworks is that Calabash tests are written in Cucumber. That means the test is written like a specification and is simple and easy to read even for non-tech people, but still executable by the automation system.

CALABASH CODE SAMPLE

Feature: Answer the Question feature
Scenario: As a valid user I want to answer app question
I wait for the text “What is the best way to test application on hundred devices?”
Then I press Radio button 0
Then I press Radio button 1
Then I press Radio button 2
Then I enter text “Simple Test” into field with id “editText1”
Then I press view with id “Button1”

EARLGREY

To some degree, EarlGrey is the ‘Espresso for iOS’. It’s also developed and open sourced by Google. Google uses this test framework to test many iOS native apps including Google Calendar, YouTube, etc. As the codename goes, lots of similarities can be found between Espresso and EarlGrey. For example, EarlGrey tests will automatically wait for events (animations, network requests etc.) before trying to interact with the UI.

EARLGREY CODE SAMPLE

– (void)testBasicSelectionAndAction {
[[EarlGrey selectElementWithMatcher::grey_accessibilityID(@”ClickHere”)]
performAction:grey_tap()];

// Example of long press with EarlGrey matchers
– (void)testLongPress {
[[EarlGrey selectElementWithMatcher::grey_accessibilityLabel(@”Box”

)]
performAction:grey_longPressWithDuration(0.5f)];
[[EarlGrey selectElementWithMatcher::grey_accessibilityLabel(@”One Long Press”)]
assertWithMatcher:grey_sufficientlyVisible()];

// Example of multi-select, visible click on items
– (void)testCollectionMatchers {
id visibleSendButtonMatcher =
grey_allOf(grey_accessibilityID(@”Box”), grey_sufficientlyVisible(), nil);
[[EarlGrey selectElementWithMatcher:visibleSendButtonMatcher]
performAction:grey_tap()];
}

JEST / JASMINE

Jest uses Jasmine behavior-driven framework as the basis for testing JavaScript code. Every test case starts from describe() function call, similar to how JUnit uses TestCase class. The describe() function takes 2 parameters – the description/title of the test case and the function to be executed. The it() function includes all the test steps and provides (similar to JUnit) series of expect() functions.

JASMINE CODE SAMPLE

describe(“Player”, function() {
var player;
var song;
beforeEach(function() {
player = new Player();
song = new Song();
});
it(“should be able to play a Song”, function() {
player.play(song);
expect(player.currentlyPlayingSong).toEqual(song);
//demonstrates use of custom matcher
expect(player).toBePlaying(song);
});
describe(“when song has been paused”, function() {
beforeEach(function() {
player.play(song);
player.pause();
});

it(“should indicate the song is paused”, function() {
expect(player.isPlaying).toBeFalsy();
// demonstrates use of ‘not’ with a custom matcher
expect(player).not.toBePlaying(song);
});
it(“should be possible to resume”, function() {
player.resume();
expect(player.isPlaying).toBeTruthy();
expect(player.currentlyPlayingSong).toEqual(song);
});
});
// demonstrates use of spies to intercept and test method calls
it(“tells the current song if the user has made it a favorite”, function() {
spyOn(song, ‘persistFavoriteStatus’);
player.play(song);
player.makeFavorite();
expect(song.persistFavoriteStatus).toHaveBeenCalledWith(true);
});
//demonstrates use of expected exceptions
describe(“#resume”, function() {
it(“should throw an exception if song is already playing”, function() {
player.play(song);
expect(function() {
player.resume();
}).toThrow(“song is already playing”);
});
});
});

BONUS: UI AUTOMATION

UI Automation is a tool developed by Apple that automates the testing of iOS applications. Put in another way, UI Automation to iOS is similar with UI Automator to Android. UI Automation tests are written in JavaScript, adhering to an API defined by Apple. Below is the code written on Apple’s Developer portal.

NOTE! UI Automation still works fine if you are some using older version of Xcode, but with the latest Xcode, the entire UI Automation framework is deprecated by Apple.

UI AUTOMATION CODE SAMPLE

UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0]
If Edit button is the first button in your code, you can also have the code as below:
UIATarget.localTarget().frontMostApp().mainWindow().buttons()[“Edit”]
Then you can use any of these three ways to call an action to tap that button:
UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0].tap();
UIATarget.localTarget().frontMostApp().mainWindow().buttons()[“Edit”].tap();
var editButton=UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0];
editButton.tap();
The Automation instrument has a complete element hierarchy that represents your app’s user interface. To view that hierarchy, use the logElementTree method to write an outline of it to the log:
UIATarget.localTarget().frontMostApp().logElementTree()

CONCLUSION

Though iOS app testing is totally different from Android app testing, you can use either Appium or Calabash to create test scripts that can be used to do cross-platform testing. That being said, every framework has its advantages and requirements are unique for each project. It’s recommended to do some further research, learn more about each framework and choose the one you are comfortable with. Happy iOS testing.

Top 5 Android Application Testing Framework

Did you know?
There are 300+ Million Android devices across the globe
850,000 daily Android activations
Google play has over 450,000 Android Apps available

Take a look few key App Highlights to discuss further

Google’s Android ecosystem is continuing to expand rapidly. It is evident that new mobile OEMs are emerging in every corner of the world adds in different screen sizes, ROM/firmware, chipsets, etc. For Android developers/testers, it becomes relatively difficult to cope up with the fragmentation.

Luckily, Android developers/Testers have unlimited access to some advanced cloud-based mobile testing solution, like Testdroid Cloud, for the Large-Scale Automation of real devices for quality assurance.

Also, the emergence of different Android testing frameworks has substantially relieved Android developers’/Tester lives.

In this blog, we are going to discuss five most used Android testing frameworks and break down the basics and code examples of each.

ROBOTIUM

Definitely, Robotium was once the most widely used Android testing framework in the early days of the Android world. It is similar to Selenium in Android and makes testing API simpler.

Robotium is an open source library extending JUnit with abundant useful testing methods for Android UI testing. It offers powerful and robust automatic black-box test cases for Android apps (native and hybrid) and web testing. With Robotium we can write function, system and acceptance test scenarios, and test applications where the source code is available.

ROBOTIUM CODE EXAMPLE

// Public void for the operation
public void testRecorded() throws Exception {
// Wait for the text ‘Hello!’ to be shown for newbie
if (solo.waitForText(“Hello!”)) {
// R class ID identifier for ‘Sign in’ – and click it
solo.clickOnView(solo.findViewById(“com.twitter.android.R.id.sign_in”));
// R class ID identifier for entering username
solo.enterText((EditText) solo.findViewById(“com.twitter.android.R.id.login_username”),”username”);
// R class ID identifier for entering password
solo.enterText((EditText) solo.findViewById(“com.twitter.android.R.id.login_password”),”password”);
// R class ID identifier for clicking log in
solo.clickOnView(solo.findViewById(“com.twitter.android.R.id.login_login”));
// Wait until log in is done
solo.waitForActivity(“HomeTabActivity”);
}
// Activate the text field to compose a tweet
solo.clickOnView(solo.findViewById(“com.twitter.android.R.id.menu_compose_tweet”));
// Type the tweet
solo.enterText((EditText) solo.findViewById(“com.twitter.android.R.id.edit”), “Testdroid”);
// Tweeting!
solo.clickOnView(solo.findViewById(“com.twitter.android.R.id.composer_post”));
}

For convenience, Testdroid Recorder is used. It is an amazing recording tool built with Robotium for test script creation. By performing actual actions on the real device, the tool records every step or action taken and converts it to Javascript for further modification.

In addition to that, it is also entitled to download and utilize the extension Library – ExtSolo.

It includes useful methods that have not been merged into Robotium, for instance:

• Automatic scaling of clicks for any resolution
• Multi-path drags
• Automatic test failure screenshots
• Mock locations
• Change device language
• WiFi control

UIAUTOMATOR

Although Robotium is a good yet basic framework, UIAutomator offers Android apps and games testing. Google’s test framework allows user interface (UI) testing of native Android apps on one or more devices. Another advantage of UIAutomator is that it runs JUnit test cases with special privileges. This means that the test cases can span across different processes. It also provides five different classes to use, including

UiCollection | UiDevice | UiObject | Uiscrollable | UiSelector

But, it only works on Android devices with API level 16 or higher. Another downside of UIAutomator is that the tool doesn’t support web view, with no way for direct access to Android objects.

UIAUTOMATOR’S CODE EXAMPLE

// Public void for the operation
public void testSignInAndTweet() throws Exception {
// Starting application:
getUiDevice().wakeUp(); // Press Home button to ensure we’re on homescreen
getUiDevice().pressHome(); // Select ‘Apps’ and click button
new UiObject(new UiSelector().description(“Apps”)).click(); // Select ‘Twitter’ and click
new UiObject(new UiSelector().text(“Twitter”)).click(); // Locate and select ‘Sign in’
UiSelector signIn = new UiSelector().text(“Sign In”); // If button is available, click
UiObject signInButton = new UiObject(signIn);
if (signInButton.exists()) {
signInButton.click(); // Set the username
new UiObject(new
UiSelector().className(“android.widget.EditText”).instance(0)).setText(“username”);
new UiObject(new
UiSelector().className(“android.widget.EditText”).instance(1)).setText(“password”);
new UiObject(new UiSelector().className(“android.widget.Button”).
text(“Sign In”).instance(0)).click(); // Wait Sign in progress window
getUiDevice().waitForWindowUpdate(null, 2000); // Wait for main window
getUiDevice().waitForWindowUpdate(null, 30000);
}
new UiObject(new UiSelector().description(“New tweet”)).click(); // Typing text for a tweet
new UiObject(new UiSelector().className(“android.widget.LinearLayout”).instance(8)).
setText(“Awesome #Testdroid!”); // Tweeting!
new UiObject(new UiSelector().text(“Tweet”)).click();

ESPRESSO

Espresso is the latest Android test automation framework that got open-sourced by Google. It is now available for developers and testers to beat out the UIs. Espresso has an API which is small, predictable, easy to learn and built on top of the Android instrumentation framework. Also, Android UI tests can be quickly written with it. It supports API level from 8 (Froyo), 10 (Gingerbread), and 15 (Ice Cream Sandwich) and afterwards.

It’s quite reliable, synchronizing with the UI thread is fast, because, there is no need for any sleeps (test run on same millisecond when an app becomes idle).
But it does not support web views.

ESPRESSO CODE EXAMPLE

public void testEspresso() {
// Check if view with the text ‘Hello.’ is shown
onView(withText(“Hello.”)).check(matches(isDisplayed()));
// R class ID identifier for ‘Sign in’ – and click it
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier(“com.twitter.android:id/sign_in”, null, null))).perform(click());
// R class ID identifier for entering username
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier(“com.twitter.android:id/login_username”, null, null))).perform((typeText(“username”)));
// R class ID identifier for entering password
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier(“com.twitter.android:id/login_password”, null, null))).perform((typeText(“password”)));
// R class ID identifier for clicking log in
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier(“com.twitter.android:id/login_login”, null, null))).perform(click());
// Activate the text field to compose a tweet
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier(“com.twitter.android:id/menu_compose_tweet”, null, null))).perform(click());
// Type the tweet
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier(“com.twitter.android:id/edit”, null, null))).perform((typeText(”#Testdroid”)));
// Tweeting!
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier(“com.twitter.android:id/composer_post”, null, null))).perform(click());
}

CALABASH

Calabash is a cross-platform test automation framework for Android and iOS native and hybrid applications. Calabash has easy-to-understand syntax, which enables even non-technical persons to create and execute automated acceptance tests for apps on both of these mobile platforms. Calabash’s tests are first described in Cucumber and then it is converted to Robotium or Frank in run time. It supports almost 80 different natural language commands (controllers), and new controllers can be implemented to it in Ruby or Java.

CALABASH CODE EXAMPLE

Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text “Hello”
Then I press view with id “Sign in”
Then I enter text “username” into “login_username”
Then I enter text “password” into “login_password”
Then I wait for activity “HomeTabActivity”
Then I press view with id “menu_compose_tweet”
Then I enter text “Testdroid” into field with id “edit”
Then I press view with id “composer_post”

APPIUM

Appium is a mobile test automation framework (and tool) for native, hybrid and mobile-web apps for iOS and Android. It uses JSONWireProtocol to interact with iOS and Android apps using Selenium’s WebDriver. It supports Android via UIAutomator (API level 16 or higher) and Seledroid (API level lower than 16), iOS via UI Automation, and mobile web as Selenium driver for Android and iOS. Looking for a complete

One of the biggest advantages of Appium is that you can write Appium scripts can be written in almost any programming language (e.g. Java, Objective-C, JavaScript, PHP, Ruby, Python or C#, etc), freedom to select tools, compatibility across the most important platforms (Android and iOS), freedom to install and configure devices to test. Also, familiarity with Selenium makes it easy to use Appium in mobile app testing. They use the same WebDriver and the desired capabilities are also used in the same way. Configuring an application to run on Appium has a lot of similarities to Selenium.

APPIUM CODE EXAMPLE

# wait for hello
sleep(3)
textFields = driver.find_elements_by_tag_name(‘textField’)
assertEqual(textFields[0].get_attribute(“value”), “Hello”)
# click sign-in button
driver.find_elements_by_name(‘Sign in’)[0].click()
# find the text fields again, and enter username and password
textFields = driver.find_elements_by_tag_name(‘textField’)
textFields[0].send_keys(“twitter_username”)
textFields[1].send_keys(“passw0rd”)
# click the Login button (the first button in the view)
driver.find_elements_by_tag_name(‘button’)[0].click()
# sleep
sleep(3)
# click the first button with name “Compose”
driver.find_elements_by_name(‘Compose’)[0].click()
# type in the tweet message
driver.find_elements_by_tag_name(‘textField’)[0].send_keys(”#Testdroid is awesome!”)
# press the Send button
driver.find_elements_by_name(‘Send’)[0].click()
# exit
driver.quit()

WRAPPING UP

Here are top 5 android testing frameworks for daily Android builds, creation, and correction. Certainly, each of them has its own pros and cons.

Appium is good in testing both your Android and iOS versions at the same time. If it is for a loyal Android developer with only Android-version app, for instance, then using Robotium is a good idea too.
Also, Testdroid Recorder will definitely save lots of time and money and It’s free. Therefore, think about testing need – functional testing, compatibility testing, UI testing, etc. – and select the right and best Android testing framework(s).

Thanks for Reading!