Mobile Testing with Simulators/Emulators vs Real Devices

Mobile testing has become a nightmare due to increasing number of devices and configurations that mobile apps and websites need to be cross-checked against. These tests should confirm the seamless running of applications. Thankfully, today we have a wide range of testing tools for thorough testing.

These tools can be categorized into three broad categories: emulators, simulators, or real devices.

Well, few may say that emulators and simulators can be clubbed in a single group. But, real device testing does what the name implies – it tests the application as they run on the users’ smartphones.

Let’s take a deeper look at the categories.

Real device mobile testing requires the use of the actual smartphone. But emulators and simulators are based on the concept of virtual testing. Virtual testing involves testing the software that provides the same functionality as provided by the real phone.

For testing mobile apps, an emulator is a desktop application that mimics the hardware and OS of the applications that should be tested. A simulator does not mimic the hardware/OS, but rather it mimics the basic behavior of a device. While simulators are usually simpler in function than emulators, they are not as useful as emulators. Real device testing checks the functionality of mobile apps thoroughly and ensures the appropriate working of the apps.

Emulators provide better results compared to simulators as they can be used to test specific situations or cases, and also can mimic multiple devices. Emulators are often used more as they are relatively cheaper – specifically if compared to real devices.

Emulator/Simulators VS Real Devices

Let us try to compare what provides better benefits in the following scenarios:

• Situation-based scenarios: In situation-based scenarios, buying required mobile devices may be tough – while using the emulator/simulator may take care of the need. This means, not all the scenarios can be tested in simulators/emulators – this is possible only if testing is done on the mobile.
• Ease-of-use scenarios: Emulators or Simulators make testing the plethora of apps on the multiple mobile devices available today much easier, but they miss out the UI and UX aspects of an app, including the color and brightness display.
• Ease-of-access scenarios: Testing on mobile devices is reliable – however, testing on Emulators and Simulators makes it easier as it requires only the URL for the app.
• Validating battery scenarios: Mobile devices can perform this test as required whereas emulators and simulators usually are not able to mimic the issues related to the battery.
• Validating Performance: This is a very crucial part of testing a mobile application. Testing the performance of an app on a mobile device usually gives a more accurate result as compared when tested on emulators or simulators.

Which is the better option?

While both Emulators and Simulators look like great alternatives for testing apps, since having a wide range of real devices also accounts for the cost involved. But, they have issues that may generate both false positive and negative results, which will surely have a negative impact on the business ROI. Another disadvantage of emulator/simulator is that howsoever much close their feature may be to the real app, emulator/simulator may still not cover all features/scenarios and the results obtained may not be as accurate as we would want them to be.

Real device testing wins the challenge over either of the two. With an efficient testing process, real devices can validate for every possible scenario with almost 100% accuracy.

However, having all three options have their own benefits, and can be used according to the need, helping the organizations meet their goals.

Subscribe to read more on Mobile Test concepts, tools, and strategies.

Regression Testing for a Game Software

We have discussed a lot about the video game market. It is forecasted that the global games market will reach up to $ 108 Billion with a mobile gaming percentage of 42%.

The software with the largest number of tweaks, releases, updates, and version are games. Video games have to be updated in line with the customer sentiments and satisfaction. Thus, Gaming Testing is a segment where regression plays a very crucial role.

Major challenges faced while testing a game software are:

1. Almost all games are based on mathematics, hence a focus should be taken to solve them for testing. It involves a huge amount of time when regression testing proves to be an ice breaking solution.
2. Maintaining deliverables for all the possible game versions, releases and updated is a very difficult task.
3. Thinking like an end user of different age category and finding bugs thinking from the user’s perspective is a very difficult task.
4. Time to market is an essence to a game’s success. Testing multiple game version at the same time becomes a very difficult task for the game testing team.

Steps for Regression Testing of a Game Software

Step 1: Smoke and Sanity  Sounds familiar?? But smoke and sanity testing are essential to make sure the system or the platform under test is under desirable conditions. The main purpose is to ensure if the system is stable even before the game testing process is initiated.

Related:  Read our blog on Smoke Testing vs Sanity Testing

Step 2: Requirements Analysis Testing requirements of the modifications or updates to the code must be meticulously analyzed. Often users encounter bugs that are found to be a result of last-minute alterations which can be found in requirement analysis. Mandatory requirements for game testing must be carefully assessed, and test cases for regression should be prepared such that the core features of the game software remain firmly intact.

Step 3: Identifying Test Cases for Critical Game Functions Of the various test cases designed for game regression testing, the most critical for gamers and development teams alike are the Sanity test cases that check the basic functionality of the gaming system. In this, ordinary setup related test cases are then tested on priority. Then, the test cases that are designed for regression testing are executed, as per the bandwidth and the requirement. Integration testing and its test cases are highly important and there needs to be a series of regression test cases especially while performing integration testing. A sudden last moment bug fix, for example, can break the integration between multiple modules, even in the tested game applications.

Step 4: Test Case Selection After prioritizing the test cases, they are selected for regression test execution. The selection of these test cases is basically done in the area of frequent changes and based on the features and their criticality. Tests are run aggressively for those units of code that have undergone multiple changes repeatedly.

We will discuss more Regression Testing in our upcoming blogs.

Test Automation in Large Scale

Digital businesses must adapt to new functionalities into production on daily basis to take full advantage of the flexibility offered by the virtual world.

Automation for digital business is not science fiction. It’s real and happening today. For one of the largest U.S. Federal programs, more than half a million business process were automated every day in 2 hours on more than 100 virtual machines.

In another example, a global manufacturer of luxury brands validates their core business processes worldwide with 600 hours of automation on more than 30 virtual machines every day!

We can see every day that today’s digital technology projects take place on a swift timeline than ever before, where typical projects last days-to-weeks rather than months-to-years. Large-scale automation makes it possible to deploy this new business functionality early and often – while significantly mitigating risks of business disruption or major glitches during the process.

The real world scenario explained in three words

FAST                FREQUENT                  FLAWLESS

The quality of application is determined by its swiftness, frequency of updates and defects

Automation is the key. Test Automation for process validation and functional testing allows you to run through literally millions of business process steps just like the end users, to validate that no code is broken.

Though test automation does not look a very rewarding or a feasible option due to the costs involved in setting up the environment or the tools – but definitely, in the long run, its benefits surely surpass the initial glitches.

It is said, that “there’s a method for every madness”.

To maximize the ROI using Test Automation, a specific plan of action is required.

What actually might set up the best way of implementing large scale test automation so as to maximize ROI?

1. Robust Product Road Map: When organizations begin to automate anything, they should ensure that they have a clear goal set. There should be absolute clarity on the product road map and on the automation check list. Else, there will be confusion and chaos around the whole process and the efforts prove to be a big scale failure. Hence, defining the product roadmap is a must.
2. Optimized Test Process: To meet a defined roadmap, a clear demarcation of the test process is a must. The test process must be defined keeping in mind the risks, costs, and training involved while implementing test automation.
3. The Framework: Whether it is a data-driven or a keyword-driven framework, or a hybrid one, defining and selecting a framework will have a definite positive impact on the ROI as it will implement the road map you defined earlier.
4. Tools to be used: Though testers have a wide variety of choice for testing solutions, wrong decisions regarding the test automation tools can lead to effects that may not be reversible. Therefore, selecting the right tools is a business-critical requirement for ensuring successful test runs. All variables such as integration, installation, cost, maintenance, and compatibility with the testing environment must be considered while selecting a tool. 
5. Script Management: The scope of script management usually floats around the areas of the standard process that are well documented, logging the errors that are generated, and most importantly testing if the script was written is able to withstand unexpected behaviors.
6. Manual Effort: Automation is an end result of a lot of manual activities such as script and test case writing, setting up the machines and selecting the tests, and once the test is run – analyze the results. These cannot be done in a jiffy, and need to be planned for both costs and efforts.
7. Testing Team: If an organization realizes the significance of the manual effort required, it must give a due diligence while planning the work and effort estimation with regards to the team and the skills required for the implementation of test automation.
8. True Measure of Success: A few good examples of this definition may include measuring benefits such as faster time to market, improved ROI, or, on the other hand, the number of bugs have decreased. 

Scaling up with virtual machines

The automation framework can be run on masses of virtual machines in the cloud to get scale. During automated business process validation, each of these machines interacts with the applications, running through all complex business scenarios with real time data. This verifies that things are working as expected. The machines will only be turned up on demand and can be decommissioned when the scenarios finish executing.

Please share your thoughts on test automation. Subscribe to get notifications on our upcoming test automation blogs.

Software Testing Services

‘Making a Right Choice’

“Global Pure Play Software Testing Services Market 2017-2021”

Technavio’s analysts forecast the global pure play software testing services market to grow at a CAGR of 19.41% during the period 2017-2021 and is expected to reach  USD 9.08 billion by 2021.

In today’s technology-driven business environment, software apps and products are expected to satisfy critical checkpoints, such as ease of installation, performance, compatibility, recovery after a failure, resource consumption, portability, security, and reliability.

The global customer acceptance and deeper implementations of Cloud, Big Data, Mobile, and Gaming demand immediate attention and a redefined application development. When there is a new upgrade or deployment for the existing business needs, the IT enterprises should make sure that the implementation is accurate and secure. Nevertheless, it is very difficult to establish an equal level of intensive care when it comes to the everyday functioning of the business application.

This has set a higher standard for software testing services companies. They are catering to these problems with their holistic testing strategy and framework to test and certify the quality of the application and make sure that it is defect free.

The Cost of Software Testing is primarily affected by the reasons mentioned in the image below

Cost is almost always the predominant problem that IT decision-makers are worried about. Skyrocketing costs are usually the result of many fundamental difficulties that can be defined by many braces, including:

• Inefficient processes
• Underproductive people
• Inadequate tools and technologies
• Dynamic and unstable business and technological landscape
• Poor decision-making or implementation errors

This inability of an organization is resulting in catastrophic application failures where not only is money being misspent, but reputations of otherwise smart professionals are being smudged.

Adding to these challenges, another big challenge for the senior executives is gaining access to the right testing tool.

Along with good testing processes and people, testing tools are an integral part of the trio that boosts the speed and quality of software testing programs. Testing tools allow you to leverage the testing servers to create, maintain and run testing scripts, both manual and automated. A Good software testing tool provides a turn-key solution that allows to focus on creating robust and efficient tests, as quickly as possible. Such tools provide the extra vigour that is required to execute successfully. This is when software testing services company comes into the picture.

These are independent technology company who take up outsourced software testing projects.

Indium Software is an Independent Software Testing Services Company with a multi-domain focus, fostered by IP-led innovation 

There primary objective to service the global technology customers in terms of Software Testing. These services are applicable across industry sectors such as retail, e-commerce, banking, gaming, technology, education, manufacturing, life sciences, healthcare and travel/transportation/hospitality.

Software testing services include

• Test Automation
• Installation/Configuration Testing
• Regression Testing
• Mainframe Testing
• Continuous Integration Testing
• Managed Crowd Testing
• ETL Testing
• Accessibility Testing
• Exploratory Testing
• Internationalization/Localization Testing
• Automated Test Script Development
• User Acceptance Testing (UAT)

• Smoke and Sanity Testing
• Compatibility Testing
• Security Testing
• Functional Testing (Unit, Integration & System)
• Performance Testing
• Defect Verification
• GUI and Usability Testing
• System/Integration Testing
• Alpha/Beta Testing
• Independent Verification & Validation
• Industry-specific testing

Benefits of Partnering with Software Testing Services Company

• Streamlined software testing process
• QA Testers can optimize the QA system
• Generates high-quality software projects, mobile-ready apps, and market-ready service deployments with lesser time to market.
• Cost-benefit realities of QA effort to enhance the economic efficiency of the business
• Improve the current QA thrust and improve it through automation and workflows
• QA services ensures High RoI on QA investments
• Provides 100% test coverage
• Provides faster test cycle at reduced cost

Read our blog on  Advantages of Software Testing Outsourcing

We will discuss more about software testing services in our upcoming blogs.

Till then, stay tuned !!!!

Mobile Application – Performance Testing & Engineering

Why is performance testing important for mobile apps?

Why is performance testing important and why should you be doing it? Well, very simply, it is the key to user engagement, getting people to use your app and continue using your app.

User engagement/experience is definitely a challenge on mobile. This graph is based on data from Apple and Google shows that 80% of apps are never used again after the first day they are downloaded and less than 4% are still used a month after they are downloaded. Looking at the graph, it’s evident that keeping users engaged with your mobile app is difficult.

A side point is that I have had people in the past say that while they accept this for consumer-facing apps, it’s not relevant to them because they are making employee facing apps or some other B2B type apps. But the reality is that adoption and engagement are just as much a challenge in employee facing apps as it is in B2B apps.

Going a bit further I would like to illustrate with two examples with data points, which basically impacts the impacts of performance of user engagement and ultimately the revenue

• Amazon has reported that a 100-millisecond increase in response time of their app reduces revenue by 1%.
• Google reported similar findings that a 100-400 milliseconds increase in search response time led to a $90 million dollar reduction in ad revenue.

It’s important to note that there’s nothing unique about Amazon or Google, meaning that this impacts them. So, clearly performance is critical to users, so we need to be testing it.

Importance of Mobile Apps Performance

As the usage of mobile phones is increasing, the importance of mobile apps performance is also increasing simultaneously.  Mobile phone users are very demanding and always expect mobile apps to perform like their desktop apps.

According to a survey,

• 71% users expect a website to open on their mobile as quickly as on their desktop.
• 5 seconds is considered as turnaround time for mobile applications and websites.
• 74% users leave the mobile websites and 50% users exit mobile apps after 5 seconds or more response time.
• 35% of the frustrated users go to competitors’ applications.

Normally mobile phone users try to access the troubled application twice and nearly half of them never return to the application if they still face the same issue(s) on their mobile device. Achieving and maintaining a good response time of a mobile application is more challenging for performance engineers than a desktop application due to its complex architecture and fewer resources it uses.

Difference between Traditional Desktop Web Apps & Mobile Apps

The both Desktop web applications and mobile applications are quite similar technologies and don’t need to select a unique performance testing tools for mobile apps.

Top 5 Mobile App performance related issues

Client Side	Server Side
Network (Wi-Fi-/2G/3G/4G) impact of connectivity issues CPU Usage Memory Usage 2d & 3d Graphics Card Battery Usage consumption	CPU Usage Memory Usage Cache I/O Bandwidth Usage Connections

Top 3 Performance issues which we have encountered while carrying load testing of mobile based application and solution

S.No	Issue	Solution
	Bandwidth Consumption issue
1	During our mobile app performance, test execution network bandwidth was high due to image & page size was huge	Bandwidth consumption decreased by compressing and reducing page & image resources to make it faster
	Response Time / Page Load issue
2	During mobile app performance test execution, the page load time was high.	Reducing the number of HTTP requests between the client and server.
	Data Loss Issue
3	We experienced in one of our microfinance client application while filling the forms and uploading the images in mobile application and if network connection loss/drops the middle of completing all the data will lose.	Create an offline mode/data save option method in the mobile application so that whenever network drops the data won’t lose. We can resume the continued activity once the connection has been re-established. This kind of method will use for travel apps

 Conclusion

There is no choice about the exponential growth of mobile application usage in this world. To better user experience the mobile apps should be quick to respond as expected by the customer.

The Performance testing of mobile apps is not different from our traditional web/desktop applications. However, Performance engineers need to understand all types of mobile apps architecture and details completely (like Network bandwidth, screen size, processing power etc…)so that it will improve the user experience and apparently the user base & revenue.

References:

1. http://www.testplant.com/2015/04/23/mobile-apps-need-performance-testing-too/ 

2. http://www.agileload.com/agileload/blog/2013/01/14/mobile-performance-testing-overall-analysis—whitepaper

Web Services Testing: An Overview

Web Services deliver a standard mode of interoperability between software applications running across platforms and frameworks. These form base of connectivity for services that are connected together into a Service Oriented Architecture (SOA).

Further, these services communicate with each other using web services. Web services put force a standard way of integrating web-based applications using XML, Simple Object Access Protocol (SOAP), Web Services Description Language (WSDL), and UDDI open standards over an internet protocol. A Web service is offered by an electronic device to another electronic device, communicating with each other via the World Wide Web.

It is also factual that a failed web service creates chaos not only to the managers but also to the administrators who are responsible for the server maintenance. A failure also discomforts the clients who are trying to call the particular web service.

The key elements of web services are a repository, messaging, and service. Since web services are distributed over networks and applications, the testing requirements should also include interfaces. Web services are integrally susceptible to added risks in the areas of integration and interoperability.

A web service can be implemented using any programming language on any platform, on condition that a standard XML interface description called WSDL is available. A standard messaging protocol called SOAP is also used at the simultaneously. Web services often run over HTTP but may also run over other application layer transport protocols. Automating web services testing facilitates reduction of its overall testing efforts.

Web Services Testing

Web services provide the continuous connection from one software application to another over private intranets and the Internet. Web services’ testing includes functionality and loads aspects to check how a Web service performs for single client and scales as the accessing load increases. Testing enables the detection of errors, evaluation, and approval of system qualities at an earlier stage. An automated test approach, in particular, helps to efficiently repeat tests whenever needed and by improvising the time to market. In particular, test automation will be essential to a sound and efficient Web services development process, for the assessment of the functionality, performance, security, scalability, and UI of Web services.

Web Services Testing Steps are

Step 1: Generate the client /skeleton code of the web service

Step 2: Define the required test inputs

Step 3: Invoke the web service using the client/skeleton code

Step 4: Client/skeleton code generation and response verification

Step 5: Actual response Vs expected response verification

Web Services Testing Tools

There are many web services testing tools available, such as

• ManageEngine
• QEngine
• SoapUI
• TestMaker
• WebInject toolsets

These tools are built on open source tools/libraries/frameworks and hence help reduce overall costs. They help to increase automation efficiency by minimizing initial coding effort. These tools can also help to create and execute the tests quickly.

Read our Blog on Top 10 Testing Tools

In this fast paced world, a faster test cycle at lower cost is crucial to stay competitive and thus reusable test automation frameworks coupled with open source tools and technologies is a key solution to shrink test cycle time and its costs.

There are Web Services test automation frameworks available that are designed and developed by many testing service providers and adopting them reflects more effectiveness. Some of their important features & benefits are detailed below.

We will discuss more about Web Services Testing in our upcoming blogs.

Subscribe to hear more from our web services testing experts.

IoT Testing Complexities

Posted on July 25, 2017 by Prabhu

A comprehensive QA strategy is essential to cover the dimensions of IoT testing. The strategy should include the types of testing, test lab setup, testing tools and simulators/emulators that are to be deployed. Considering the practical hiccups in generating big data from the thing (device) in a testing environment, it is crucial to evaluate data simulation and virtualization methods.

Stubs can be considered as an option during early stages, whereas, data recorders can assist as an alternative at the later stages. Beyond test planning and data simulation, metrics-driven exhaustive test execution is performed to achieve a stable system. QA team can split IoT test area into two layers i.e. device interaction layer and user interaction layer. However, QA has to be performed across both the layers. It is actually easier to identify techniques and the types of testing that can be adapted to each layer to enhance the QA strategy.

The device interaction layer

This layer acts as the connectivity between the software and the hardware components of a real-time ‘IoT environment interact’. A typical example will be a Bluetooth device transmitting real-time data to a mobile device application. Sometimes, a lot of interaction testing will be done on the functional side of QA.

However, other types of testing could also be required. IoT testing will cover the spectrum of other required elements listed below, in addition to typical software testing:

Conformance with standards: These are mostly device performance traits that are precise to the devices and sensors. These attributes must be validated against the standards of the device and its communications protocol. Hardware vendors perform most of these tests, but there could be a certain domain or use-case specific requirements such as the use of such devices in an environment that was not tested.

Interoperability: The ability of different devices to support the required functionality among themselves, other external devices and implementations.

Security: With billions of sensors in the making, it’s crucial to tackle data privacy and the security concerns across the IoT ecosystem.

The following are the different types of security testing requirements:

• Identity and authentication
• Data protection
• Data encryption
• Storage data security in local
• Remote clouds

The user interaction layer:

This layer is the touch point between the thing (IoT Device) and the user. The success of the overall system depends on the seamless user experience.

Key testing areas in this layer include:

Network capability and device level tests: The specific aspects of network communication such as connectivity are validated by simulating different network modes in addition to device-level validation such as energy consumption tests, etc.

Usability and user experience: Usability and user experience are important in terms of the real-time usability; it involves both human and machine interaction and also the real-time experience that the IoT system provides.

For example, contactless payments compared with a physical card-based payment. 

The IoT services and back-end IoT environment:

While integration testing of the interfaces is a key, there is a complex data layer that comes into play.

For example, a classic IoT system boxes a complex analytical engine to ensure an exceptional user experience. This creates a QA environment to assist validation of such interface by addressing the growing data volume, velocity, and variety challenges. The front-end validation environment can be done by assembling data recorders and simulators. The service and data layer validations will involve complex simulation such as the generation of millions of sensor hits, machine learning algorithms and the time-boxed traffic. There are a few methods to create such an ecosystem; For example, leveraging sandboxes of development services or creating mock environments using virtualization tools. However, numerous implementation synergies are required to establish a working set of environments for through services and back-end validation platform.

Let’s discuss more IoT testing in our upcoming blogs.

Please comment on your interested topic in IoT.