Security Testing Essentials of Cloud-Based Application

An IDC survey states that 87.5% of IT cloud computing businesses are concerned about Security Issues.

Let’s start with some greatest Cloud Security Breach instances!!!

1. Anthem’s Breach and the Ubiquity of Compromised Credentials
2. Amazon Cloud Horror – The demise of Code Spaces
3. Apple iCloud suffered the largest high-profile cloud security breach
4. Target security breach compromised up to 70 million customers’ credit card information during the holiday season
5. Home Depot suffered a similar fate with more than 56 million credit or debit cards and 53 million emails compromised

Cloud Security testing is very much crucial to assess the security level of the system hosted in the cloud. This requires ensuring ongoing defensive security controls and proactive regular assessments to check the apps ability to withstand the data breach threats.

The cloud security testing team should ensure if the cloud deployment is secure and should give actionable remediation information when it is not complying with security standards.

The team should proactively conduct, real-world security tests using the techniques used by hackers seeking to breach the data in cloud-based systems and applications.

The Five Cloud Security Testing Essentials for Consideration

Listed are the five essentials to be considered while adopting security testing strategy for cloud-based applications:

1. Scalability – The testing solution should be rapidly scalable with respect to the application while developing business needs without causing configuration and performance issues.
2. Availability – Availability of security testing teams working around the clock. This calls for strong test management via access to centralized test dashboards with features of effortless collaboration.
3. Speedy – Testing should be fast with short turnaround times and should have the ability to run parallel testing. This is required especially when most of the organizations are adopting agile methodologies.
4. Quality – The most important factor is that the testing should be able to make triaging of false positives and false negatives simple and fast. The reporting should include contextual, actionable guidance, empowering development team to resolve identified issues.
5. Cost – Agile methodologies not only require rapid testing but also require multiple iterations of security testing. These iterations should not incur undue incremental costs.

Cloud Security Testing Approach

1. Proactively verify the security aspects of the cloud-based systems and applications against current security hack techniques
2. Safely identify and validate critical cloud service vulnerabilities
3. Measure the susceptibility to SQL injection, cross-site scripting, and other web application attacks
4. Get actionable security information necessary to apply to the patches and code fixes
5. Verify security posture of systems and networks

Cloud testing activities do hold some challenges; your organization can overcome these hurdles. It’s imperious that the right software testing service provider would be able to ensure cloud security around applications, services, and data.

The 5S rule – Ways to a clutter free testing zone

An overview

In this competitive world of testing, with processes and methodologies being developed every day to achieve quality, how can one be up to date to attain the best results without compromising on anything and with no trade-offs? Quality being the ultimate goal, we should never hesitate to borrow anything from other industries that will make it better. When SCRUM a Rugby formation has made a revolution in Software Testing, why not a workplace organization methodology?

When things are going at a fast pace with bettering technology, there are still moments when we feel there is not enough time for the most important – Quality. Being organized is the only way out. If you are looking for ways to stay organized, here is the solution.

This article uses pointers from the 5S Japanese workplace organization methodology based on five Japanese words: Seiri, Seiton, Seiso, Seiketsu, and Shitsuke. These Japanese words translates to Sort, Straighten, Shine, Standardise and Sustain respectively. The intention of this article is to help one understand how to implement this 5S methodology in Software Testing.

The 5S methodology

The 5S methodology as Wikipedia defines it – “This methodology describes how to organize a work space for efficiency and effectiveness by identifying and storing the items used, maintaining the area and items, and sustaining the new order.” The aim of the 5S methodology is to keep the place clutter free, get rid of waste and have things organized for easy accessibility.

Now the parallel

Let us start drawing the parallel here. When every phase of testing is called out, there is a list of items that will immediately come to a tester’s mind and this forms the basis. This section, introduces the user to a few unconventional types of testing that when incorporated, will prove to be effective.

Sort

“Your mind is like this water, my friend. When it gets agitated, it becomes difficult to see. But if you allow it to settle, the answer becomes clear.” – Master Oogway (Kung Fu Panda).

The first step to getting organized to understand what is causing the clutter. In a workplace organization methodology if Sort means knowing what to keep and what to dispose, in software testing it would convert to knowing what is in scope and what is not. Defining the scope is very important. It will help keep things focused. Scope definition is part of planning. One should consider all aspects of the project’s life cycle from goals to deliverables.

Dependency Testing

In this testing type the team pre-examines the software to ensure impact of the new requirements. This process is very useful to identify what has to be included in scope weighing the impact of the upcoming change.

Straighten

“Our first game is called Well Begun is Half Done.” – Mary Poppins (Mary Poppins)

Know where to begin. Keep all that you need handy. In the 5S methodology, Straighten is to arrange items so they are easy to access. When it comes to testing these items translate to data. Keeping data ready at a hands distance is important to be organized in the testing approach. Once we have identified the scope, make ready the data. Sometimes, thinking of the test bed can open up avenues for recommendations.

Fuzz Data Testing

This technique involves feeding the system with a set of data to find error in coding loops and in the behaviour of the application. Automation skills will help. This testing when conducted effectively will cover several traditional tests for boundary value, equivalence partitioning, etc. and will also test the system for recovery, etc.

Shine

“Did you rub my lamp?” – Genie (Aladdin and his Wonderful Lamp)

Rub till it shines, who knows what you will uncover. Keeping the work area spic and span has always led to higher productivity. In testing, keep the virtual test space clean. Create Test Suites for identified tests and keep it focused for a build or release. This phase begins with the formal test designing – keep shining your tests by breaking them down to modular test cases.

Decision Table Testing

Take this approach to ensure test coverage. This type of testing helps lay out the possible combinations in a table so none is missed. It also reduces planning effort. From this table scenarios can be picked and converted to test cases.

Standardise

“With great power comes great responsibility” – Uncle Ben (Spider Man)

Once you get the handle of things and you feel you are in control, you have the responsibility of making it a standard. Standardise best practices – Create checklists for acceptance – Freeze templates. Simple is the key. Try to keep only what is necessary. If there are Test Management Tools, these do most of the delivery related metrics for you. So don’t worry about maintaining this data.

Yardstick Testing

To standardise, passing rules / conditions have to be formulated. Pick relevant parameters and establish benchmark for acceptance of an application for testing. Remember this is not a Smoke test; it has to be at a higher level. This testing technique needs some research and this will get better with time.

Sustain

“They haven’t complained yet.” – Captain von Trapp (The Sound of Music)

Don’t wait to hear feedback. Keep working by maintaining. Sustain means to maintain and also to keep improving. Apply / document experience gained. Further develop knowledge, skills and ability gained along this methodology and use this as input to improve.

Endowment Testing

Endowment Testing: This is a periodic analysis of the application to further build. The results of this testing can help evaluate the application’s ability to grow. Understand parallel systems in the market and recommend value added features to better the system

Taking it beyond

In some organizations, 5S became 6S – the 6th S being Safety – our parallel being Security. Once we have the basic testing in place, the methodology can be extended to include Security testing.

Wrapping up

Adapting to this methodology in Software Testing can help the testing team get organized and be prepared for upcoming requirements which is the smaller unit of any big project. There is no hard and fast rule here, the process can be tweaked to suite your need. This article tries to highlight a flow that will suite every software testing methodology though.

• A methodology that will suite any type of testing / domain / life cycle – Once the tester gets the basic idea of this system, it can be applied anywhere.
• Sets clear goals to keep only what is necessary – This enables the testers to stay focused.
• Augmentations tips that will help make each phase better – Introducing the tester to many more testing types that will improve quality.

FREE-UP the clutter and FOCUS on the most important.

Proactive Application Performance Testing to Prevent Outages

Addressing the commonly faced challenges by app developers in providing customers with a smooth and reliable user experience.

The apps market is growing in leaps and bounds and new features are being introduced almost every day to stay ahead of competition. Just to put it in perspective, the Apple App store has a base of nearly two million and Google play, over 2.2 million apps. Gartner predicts that an income of $77 billion will be generated in 2017 from over 268 billion mobile downloads. Compare this to $35 billion revenues in 2014, $45 billion in 2015, and $58 billion in 2016. The primary target is the age group falling between 18 and 24 years followed by 25 to 34 years.

Retaining customer interest and loyalty will be one of the greatest challenges for app developers as customers are spoilt for choice. Performance testing will become critical and a game changer determining who leads the pack.

The Challenges to Successful Implementation

App developers face several challenges that can hurt the performance of the app at various stages:

Three key mistakes that can turn away customers from retail / eCommerce mobile apps and website performance:

• Not optimizing the images
• Not compressing the files
• Not using the browser cache

Top 3 mobile app performance issues encountered in Cloud based load testing

• Often sites use different JavaScript libraries for Android and Apple iOS, which can get ‘out of sync’. The result is Java Script on a page that is no longer compatible with all page-types.
• Failure to incorporate the small rendering/DOM quirks between browsers on Android and iOS
• Incompatibility between third party library interactions

Top five capabilities that need to be effectively addressed to avoid performance challenges that impact the business or revenue model of the mobile application

• Ability to segment the delay in application response times as server, network, and application delay
• Ability to translate application performance metrics into business metrics such as page views, conversions, employee productivity, and customer satisfaction
• Develop capabilities for correlating application performance metrics with appropriate business metrics (customer satisfaction, revenue growth, brand reputation, etc.)
• Identify parts of the enterprise infrastructure that are causing the deterioration of business services
• Ability to measure application performance from a business user’s perspective

Understanding and assessing the business impact of the performance issues pertaining to mobile applications

• One additional second of delay beyond defined baselines for performance of Mobile / Web applications can cause loss of business revenues. Studies show end users are willing to wait typically between 4 and 6 seconds for a page to open before they abandon a session and move on to the next app / website in their search.
• Brand perception drops among existing customers while also driving away new customers, thereby increasing the churn out ratio.

A comprehensive, planned performance testing process can help developers overcome all of these challenges before the release, thus enhancing user experience. Indium’s jumpstart kits and iAccelerate automation frameworks further reduces testing time, meeting the needs of app developers to deliver on time and with quality.

Indium’s performance testing process is aimed at increasing revenues, expanding user base and growing the business.

Top Performance Testing Tools

What happens when millions of users with their unique credentials try to login at a time?
How will you forecast and prevent performance outages?

The best way this situation can be handled is through – virtual user (VU). For all our tests, the VU behaves just the way a real user would.
For the creation of as many VUs, as you would require and simulate real-time conditions, performance testing tools are employed. Not only that, Performance testing also tests for the peak load usage, breakdown point, long-term usage, etc.

To enable all with limited resources fast and to obtain reliable results, tools are often used for this process. There are a variety of tools available in the market- licensed, free wares and open sourced.

Check out our list which covers top performance testing tools (both open-source as well as licensed).

Open Source Performance Testing Tools

1.Apache JMeter

Open source software and 100% pure Java application which is designed to load test functional behavior and measure performance.

Features

• Full-featured Test IDE that allows fast Test Plan recording
• Dynamic H TML reporting
• Full multithreading framework allows concurrent sampling
• Complete portability
• Highly Extensible core

2.Grinder

The Grinder is a popular Java-based framework for load testing. Jython (the special Java implementation of Python) is the default script language. TestScripts also could be written in Clojure.

The main parts of The Grinder:

• The Grinder Console is the basic interactive development environment (IDE) for test suites developing and editing. It monitors results in real time and controls various agents of The Grinder.
• The Grinder Agents are the special headless load generators. They have a number of Workers to create the load. 

Key features of Grinder

• It is a cross-platform tool. It could run on any platform and a Java Virtual Machine can be set up immediately.
• Multiple protocols support.
• Rapid and powerful script development
• Simple, sliding performance graphs for all transactions in one tab.
• The users have an ability to write the own plug-ins to a documented interface
• Flexible parameterization.
• The user has an ability to load Java API as a test from HTTP web servers, SOAP and REST services.
• Flexibility for loading and executing third-party libraries.

3.Gatling

Gatling is the powerful Scala-based “firearm” for load testing with two executables:

• one for test case recording
• other for test case execution

The main features of Gatling are listed below

• It works with almost all operating system and any browser.
• Gatling supports colorful and informative graphical reports containing valuable information for analysis.
• It can execute its test cases in different test clouds.
• Gatling could be easily integrated with Jenkins through the Jenkins-plugin and runs the tests through Gradle and Maven with the help of Maven plugin and Gradle plugin.
• Gatling uses only a small amount of memory.
• It reduces the debugging phase time.
• Errors and drawbacks are detected in the early part of the development cycle.

4.Locust

Locust is a code-driven, distributed load testing suite built in Python. It is frequently used for load testing websites and indicating how many concurrent users a system can handle.

According to it, the swarm of locusts will attack the website during the test. This tool can define the behavior of each locust that is the user. The swarming process is monitored from a web UI in real-time and helps to identify the weak points of the code.

Locust’s key features

• Any system or platform can be tested with this tool.
• It is the cross-platform and easily extendable tool.
• Hundreds of thousands of users’ behavior can be simulated with Locust.
• Locust has reusable codes.
• It has the high level of customization. There is an ability to combine several requests as one scenario by grouping requests with specific custom points.

5.Tsung

Tsung is a distributed load testing tool which is available free of charge as an open source software product. It is protocol-independent and can be used to stress Jabber/XMPP, HTTP, SOAP, LDAP and PostgreSQL servers. The tool can simulate very large numbers of users per server, making it ideal for analyzing and testing the performance of large-scale applications, such as instant messaging solutions.

High performance and ease of use are the main features of this tool. Tsung can simulate up to 50,000 simultaneous users per physical computer, whereas traditional injection tools can usually only simulate up to 1,000 users. Based on XML configuration, Tsung removes the need for developers to write complex scripts. The tool handles tasks such as managing cookies automatically, so developers can focus on designing the scenarios not managing lines of code.

Originally known as IDX-Tsunami, Tsung was first developed in 2001 by a company called IdealX. Its current name stands for Tsunami Next Generation. Developed in the Erlang programming language, the tool is now an open source solution that is actively developed and supported by ProcessOne.

Commercial Tools for Performance Testing

1.LoadRunner

This is a HP product which can be used as a performance testing tool.  This can be bought as a HP product from its HP software division.  Also, it is very much useful in understanding and determining the performance and outcome of the system when there is an actual load. One of the key attractive features of this testing tool is that it can create and handle thousands of users at the same time. This tool enables to gather all the required information with respect to the performance and also based on the infrastructure.  The LoadRunner comprises of different tools; namely, Virtual User Generator, Controller, Load Generator and Analysis.

LoadRunner System Requirements: Microsoft Windows and Linux are the favorable OS for this measuring tool.

2.Neoload

NeoLoad is a load and stress testing tool to measure the performance of web and mobile applications. NeoLoad provides pragmatic solutions to developers to optimize the performance before the application goes into production.

Using the script less GUI and a host of automation features, NeoLoad lets you design tests 5-10x faster and maintain lesser scripts which are as little as 10% of the original design time while helping to automate testing with Continuous Integration system with the agile feature.

3.Rational Performance Tester

The Rational performance tester is an automated performance testing tool which can be used for a web application or a server-based application where there is a process of input and output is involved. This tool creates a demo of the original transaction process between the user and the web service. By the end of it, all the statistical information is gathered and they are analyzed to increase the efficiency. Any leakage in the website or the server can be identified and rectified immediately with the help of this tool. This tool can be the best option in building an effective and error free cloud computing service. This Rational Performance tester was developed by IBM (Rational software division). They have come up with many versions of this automated testing tool.

Rational Performance Tester System Requirement: Microsoft Windows and Linux AIX good enough for this performance testing tool.

4.WebLOAD

WebLoad is a Load and performance testing tool for web applications. WebLOAD allows to perform load and stress testing on any internet application using Ajax, Adobe Flex, .NET, Oracle Forms, HTML5 and many more technologies. The load can be generated load from the cloud and on-premises machines. WebLOAD’s benefits are its ease of use with features like DOM-based recording/playback, automatic correlation, and JavaScript scripting language. The tool supports large-scale performance testing with heavy user load and complex scenarios and provides clear analytics on the functionality and performance of the web application. Another highlight of the tool is its flexible licensing mechanism and pricing.

System Requirements: Windows, Linux

5.WAPT

Performance testing tool for websites and intranet applications: WAPT refers to the Web Application Performance tool.  These are scales or analyzing tools for measuring the performance and output of any web application or web related interfaces. These tools help us to measure the performance of any web services, web applications or for any other web interfaces. With this tool, you have the advantage of testing the web application performance in the various different environment and different load conditions. WAPT provides detailed information about the virtual users and its output to its users during the load testing. This is considered to be the best cost-effective tool for analyzing the performance of the web services. The WAPT tools can tests the web application on its compatibility with the browser and operating system. It is also used for testing the compatibility with the windows application in certain cases.

WAPT System Requirement: Windows OS is required for this testing tool

• Microsoft Testing Suite (VSTS)

6.SmartMeter.io

SmartMeter.io is a performance and load testing tool using a familiar JMeter interface, yet adding enterprise-level features.

SmartMeter.io makes it super simple to create a test scenario thanks to its state of the art test recorder. It lets you create test scenarios just by clicking in an embedded browser. There’s no proxy setup or browser plugin.

It also automatically generates test reports with all details about the test and its results containing graphs, statistical overview, evaluated acceptance criteria, trend analysis, and a graph comparison tool.

Among other features, it is strong in distributed testing, CI integration, and offers unparalleled performance testing support for Vaadin apps. Compatible with Windows, Linux, and macOS.

7.LoadUI

Create sophisticated load tests with just a few clicks.

With LoadUI NG Pro it is easy to model real-world loads on the API. LoadUI supports REST, SOAP, JMS, MQTT and many other API formats. The tool has an easy-to-use graphical interface which makes it simple for new users to setup load scenarios, while also providing advanced scripting features for those with more experience.

This tool allows to analyzing results than configuring and building tests by hand. The tool can also be used to report server performance data. This tool allows to setup local or globally distributed load agents for the load testing scenarios. LoadUI NG Pro allows easy reuse of existing functional test cases from SoapUI and SoapUI NG Pro.

System Requirements: Windows, Linux, and Mac OS

8.LoadComplete

SmartBear LoadComplete is a desktop tool for load, stress, scalability testing of websites and web applications. Designed from the ground-up to help you focus on the tasks at hand, it does not require advanced programming skills. It works with both traditional HTML websites and with Rich Internet Applications using modern web technologies like AJAX, ASP.NET, Java, Flash, Flex, and Silverlight. The ‘record and playback’ and ‘visual programming’ features enable you to create powerful and flexible first load test within a few seconds.

LoadComplete allows you to generate load from virtual machines, on-premise computers, or even the cloud. Customized reports in LoadComplete help you get end-to-end visibility and find problems at both client and server side.

Top Cloud-Based Performance Testing Tools

1.StromRunner Load

Simple and scalable load testing tool used to design & create mobile and web load test within a short duration of time for easy cloud-based testing without the need to schedule, deploy, and manage load generators. It creates test scripts in the browser using HPE’s patented TruClient technology, an innovative scripting tool based on node.js. StormRunner supports LoadRunner Web, Performance Center, Web Services, Flex, Mobile and TruClient protocols or it can simply upload from REST call, HAR file, CSV or import JMeter scripts.

The tool detects problems fast. It emulates and tests real-world network conditions with Network Virtualization to get accurate and repeatable results. It provides a comprehensive root cause, trending, and regression analysis as well as the intelligent and predictive analytics for a holistic client, network, and server-side breakdown report. StormRunner integrates with Jenkins, Gatling, Dynatrace, New Relic, Docker, AWS Code Pipeline and much more.

With this tool, we can Scale from 1 tester to 1,000,000 or more geographically distributed web and mobile users, add or remove them during runs.

2.BlazeMeter

With BlazeMeter, developers/testers can easily create tests (or harness existing open source scripts) in the favorite editor using straightforward YAML or JSON syntax. With this tool, one can create and edit performance tests alongside code without switching context or having to open other tools. With this keep the tests in the preferred version control repository to track changes and revert when necessary.

3.Loadstorm

Loadstorm is a cloud load testing tool for web applications. Loadstorm is the cheapest available performance and load testing tool. It has the option of creating required test plans, testing criteria and testing scenario. It can generate up to 50000 concurrent users by generating traffic to your website and then carry out the testing. This tool can bring an end to all the expensive performance testing tools. The cloud infrastructure is used in this tool, which enables you to send a huge amount of requests per second. There are thousands of servers available around the world for this software. They are proudly known as the lowest cloud load testing tool. There is no need of any scripting knowledge for using this tool. You will be provided with many graphs and reports which measure the performance in various metrics such as error rates, average response time and a number of users. This tool is available for free, but the premium account comes with added features.

System Requirement:  Windows OS.

4.CloudTest

SOASTA CloudTest is a performance testing tool for web, mobile apps and APIs. The users or the developers can use the cloud platform as their virtual testing lab. The developers can carry out the performance or load testing in the cloud platform in a cost-effective way through this tool. It has a capacity to enable user load for the website at the same time. It also increases the traffic of the website to know the actual performance under stress and heavy load.

CloudTest is not an open source tool; the price differs according to the number of load injector machines required per hour. The trial version with the power of 100 concurrent users is available for free.

Application Testing Vs Game Testing – Tug of War!!!

What Is Different About Software Testing?

Unlike game testing, software testing greatly utilizes automation scripts, and it is the matter of testing a software application, not games. These scripts can be created before or after producing written test cases, as either can be used to supplement the testing of the other similar application or program. A number of tools and frameworks have been developed to make automation quicker, as setting up automation scripts can initially take a bit of your time.

As software testing makes greater use of test cases and test scripts, this generally means that the job of a software tester is more technical than a games tester.

The field of software testing is usually seen as a more skilled profession than game testing, even though the skills required to be a games tester are exactly the same as those needed as a manual software tester.

“For software testing, you usually need a degree in computer science, software engineering, math, or anything technical, really.” 

“For games testing, a degree may be useful and a love of games, attention to detail, and a strong work ethic can usually get you into the field.”

Differences by Testing Platforms

Game Testing	Software Testing
PC, Consoles, Web/Cloud (online games likes MMOs, HTML5 and Facebook) Mobile	PC, Web, Mobile & Cloud

What Makes Game Testing Unique?

Below are the few testing types to the list of game testing checklist, which makes game testing unique.

• Fun Factor Testing
• Balance Testing
• Realism Testing
• Game Level Testing
• Game Play Testing
• Multi-player Testing
• Modification API Testing
• Game Network Testing
• Al Testing

A game tester should have the same knowledge as of software tester, but with a special focus on what makes

Game Testing is not an easy task. It requires more dedication and concentration. Now a day’s business requires for gaming consoles and game applications that are perfect from all viewpoints. For creating a good image and better response that are required to deliver games that are free from bugs and capable to run for across devices and platforms.

In games, the app is designed to give the user the least amount of access to the actions of the program as possible.

Game designers give the players with no choice but to take time to explore and discover what the game does. They want to preserve the mystery of the game world, the pursuit, the fantasy. The harder it is, and the more time it takes for the user to obtain the ‘desired results’. “Winning”.

A game is an enormously complex database or series of databases, within a shell that is built to obscure its purpose from the user.

Adding a feature of randomizer, the game design is unpredictable. Events don’t function the same way twice.

As a game tester, the interaction is only with an intentionally limiting interface that is constantly changing the way of its functions, as well as the way it looks.

“It is almost impossible to know if what is in the
screen is what is supposed to be seen”.

A game tester should be able to keep track of enormous amounts of dynamic details.

In a game’s database, it is not necessary that the same series of actions do not cause the same reaction. This makes recreating bugs extremely difficult.

Games push the technology innovation envelope. They tend to drive the hardware market, while other software conforms to it.

Gamers are often technology wizards, and are looking for the next level of performance, speed, and visual quality in their game experience. Games are always developed with an eye toward the newest, fastest, latest hardware, so game testers must keep ahead of the trends.

Testers must be educated and certified in all new hardware and have it all available in the testing environment. This technological knowledge is critical in this industry because the game developer will most often take the blame if it the game doesn’t function with the new hardware, and that failure can mean game death.

Games are more system demanding than most other software, so installation, setup, and compatibility testing is critical. Game function will be affected by video, sound, memory, and other issues more than any other type of software.

In contrast to that, the latest versions of today’s business software are released to be compatible with the standard issue in system configuration.

Conclusion

There are many similar and unique aspects between software testing and game testing but all types of testing require experts to perform it well and in a structured way.

Top iOS Testing Frameworks

Did you know that the total number of iOS devices sold in the
market is 250 Million+?
11% of the iOS users remove the app after crashes
45% of iOS users would remove the paid apps after crashes
69% of iOS users would will stop playing and remove the games after first crash
6-7 iOS devices are sold every second

With the explosion of iOS devices year on year, the verge of iOS success is increasing higher than ever before. More start-ups and SMEs don’t find it financially sustainable to acquire every piece of iOS devices with different OS versions and hardware specs. On the other hand, testing apps or games manually is not anymore an ideal option for QA process due to low efficiency and scalability issue.

Though iOS is still a more closed operating system compared to Android, many open source test automation frameworks and tools can be utilized to create robust automated tests. Along with the help of a cloud-based testing solution, it enables developers/testers to verify app quality more efficient in a scalable way and makes iOS testing tasks easier.

Today we’d like to go through 5 widely used iOS testing frameworks nowadays with simple code samples to give you a basic understanding of getting started with your iOS testing. If you are also interested in Android test automation frameworks, do not forget to check out Top 5 Android Testing Frameworks, where code samples are provided as well.

APPIUM

Appium is popular due to its flexibility and usability on both Android and iOS, and it works on native, hybrid and web applications. For iOS testing, it uses JSONWireProtocol to engage with iOS applications using Selenium WebDriver. Thanks for that, Appium does support mobile web testing very well and its use cases are very similar as if Selenium would be used for web testing.

APPIUM CODE SAMPLE

driver.findElement(By.id(“com.example.app:id/radio0”)).click();
driver.findElement(By.id(“com.example.app:id/radio1”)).click();
driver.findElement(By.id(“com.example.app:id/radio2”)).click();
driver.findElement(By.id(“com.example.app:id/editText1”)).click();
driver.findElement(By.id(“com.example.app:id/editText1”)).sendKeys(“Simple Test”);
driver.findElement(By.name(“Answer”)).click();
// or alternatively with
driver.findElement(By.id(“com.example.app:id/button1”)).click();

XCTEST / KIF

XCTest is tightly coupled with Xcode but is still usable with both real iOS devices and simulators. XCTest allows developers to write tests for components at any level and also provides the framework for UI testing capabilities. XCTest tests are grouped into subclasses of XCTestCase. Writing any tests with XCTest should be trivial to iOS developers as XCTest is fully compatible with both Objective-C and Swift.

KIF (Keep It Functional) is an iOS integration test framework that is closely related to and uses XCTest test targets. KIF tests can be executed directly in XCTestCase or any subclass. KIF allows for easy automation of iOS applications by leveraging the accessibility attributes that the OS system makes available for those with visual disabilities.

CODE SAMPLE WITH OBJECTIVE-C

– (void)testClicksOnRadioButtons {
[tester tapViewWithAccessibilityLabel:@”Radio1”];
[tester tapViewWithAccessibilityLabel:@”Radio2”];
[tester tapViewWithAccessibilityLabel:@”Radio3”];
[tester enterText:@”Simple Test”
intoViewWithAccessibilityLabel:@”editText1”];
[tester tapViewWithAccessibilityLabel:@”Answer”];
}

CODE SAMPLE WITH SWIFT

testClicksOnRadioButtons() {
let app = XCUIApplication()
app.radiobutton[0].tap()
app.radiobutton[1].tap()
app.radiobutton[2].tap()

app.staticTexts[“Simple Test”]
app.button[0].tap()
}

Learn and get started with KIF by reading our blog about this: How to get started with KIF?

CALABASH

Calabash is another great cross-platform framework that works perfectly with Android and iOS apps. One of the major differences to other frameworks is that Calabash tests are written in Cucumber. That means the test is written like a specification and is simple and easy to read even for non-tech people, but still executable by the automation system.

CALABASH CODE SAMPLE

Feature: Answer the Question feature
Scenario: As a valid user I want to answer app question
I wait for the text “What is the best way to test application on hundred devices?”
Then I press Radio button 0
Then I press Radio button 1
Then I press Radio button 2
Then I enter text “Simple Test” into field with id “editText1”
Then I press view with id “Button1”

EARLGREY

To some degree, EarlGrey is the ‘Espresso for iOS’. It’s also developed and open sourced by Google. Google uses this test framework to test many iOS native apps including Google Calendar, YouTube, etc. As the codename goes, lots of similarities can be found between Espresso and EarlGrey. For example, EarlGrey tests will automatically wait for events (animations, network requests etc.) before trying to interact with the UI.

EARLGREY CODE SAMPLE

– (void)testBasicSelectionAndAction {
[[EarlGrey selectElementWithMatcher::grey_accessibilityID(@”ClickHere”)]
performAction:grey_tap()];

// Example of long press with EarlGrey matchers
– (void)testLongPress {
[[EarlGrey selectElementWithMatcher::grey_accessibilityLabel(@”Box”

)]
performAction:grey_longPressWithDuration(0.5f)];
[[EarlGrey selectElementWithMatcher::grey_accessibilityLabel(@”One Long Press”)]
assertWithMatcher:grey_sufficientlyVisible()];

// Example of multi-select, visible click on items
– (void)testCollectionMatchers {
id visibleSendButtonMatcher =
grey_allOf(grey_accessibilityID(@”Box”), grey_sufficientlyVisible(), nil);
[[EarlGrey selectElementWithMatcher:visibleSendButtonMatcher]
performAction:grey_tap()];
}

JEST / JASMINE

Jest uses Jasmine behavior-driven framework as the basis for testing JavaScript code. Every test case starts from describe() function call, similar to how JUnit uses TestCase class. The describe() function takes 2 parameters – the description/title of the test case and the function to be executed. The it() function includes all the test steps and provides (similar to JUnit) series of expect() functions.

JASMINE CODE SAMPLE

describe(“Player”, function() {
var player;
var song;
beforeEach(function() {
player = new Player();
song = new Song();
});
it(“should be able to play a Song”, function() {
player.play(song);
expect(player.currentlyPlayingSong).toEqual(song);
//demonstrates use of custom matcher
expect(player).toBePlaying(song);
});
describe(“when song has been paused”, function() {
beforeEach(function() {
player.play(song);
player.pause();
});

it(“should indicate the song is paused”, function() {
expect(player.isPlaying).toBeFalsy();
// demonstrates use of ‘not’ with a custom matcher
expect(player).not.toBePlaying(song);
});
it(“should be possible to resume”, function() {
player.resume();
expect(player.isPlaying).toBeTruthy();
expect(player.currentlyPlayingSong).toEqual(song);
});
});
// demonstrates use of spies to intercept and test method calls
it(“tells the current song if the user has made it a favorite”, function() {
spyOn(song, ‘persistFavoriteStatus’);
player.play(song);
player.makeFavorite();
expect(song.persistFavoriteStatus).toHaveBeenCalledWith(true);
});
//demonstrates use of expected exceptions
describe(“#resume”, function() {
it(“should throw an exception if song is already playing”, function() {
player.play(song);
expect(function() {
player.resume();
}).toThrow(“song is already playing”);
});
});
});

BONUS: UI AUTOMATION

UI Automation is a tool developed by Apple that automates the testing of iOS applications. Put in another way, UI Automation to iOS is similar with UI Automator to Android. UI Automation tests are written in JavaScript, adhering to an API defined by Apple. Below is the code written on Apple’s Developer portal.

NOTE! UI Automation still works fine if you are some using older version of Xcode, but with the latest Xcode, the entire UI Automation framework is deprecated by Apple.

UI AUTOMATION CODE SAMPLE

UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0]
If Edit button is the first button in your code, you can also have the code as below:
UIATarget.localTarget().frontMostApp().mainWindow().buttons()[“Edit”]
Then you can use any of these three ways to call an action to tap that button:
UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0].tap();
UIATarget.localTarget().frontMostApp().mainWindow().buttons()[“Edit”].tap();
var editButton=UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0];
editButton.tap();
The Automation instrument has a complete element hierarchy that represents your app’s user interface. To view that hierarchy, use the logElementTree method to write an outline of it to the log:
UIATarget.localTarget().frontMostApp().logElementTree()

CONCLUSION

Though iOS app testing is totally different from Android app testing, you can use either Appium or Calabash to create test scripts that can be used to do cross-platform testing. That being said, every framework has its advantages and requirements are unique for each project. It’s recommended to do some further research, learn more about each framework and choose the one you are comfortable with. Happy iOS testing.