Max is a hardworking and qualified professional. When asked to prepare a presentation for an important upcoming meeting with potential clients, Max worked tirelessly to create a thorough and visually appealing digital presentation that could help his company close the deal. However, when it came time to do the presentation in the conference room, what was once a sleek presentation was now mediocre. Not only were the graphics too small to read, but also the color of the text were nearly impossible to read against the presentation's background. How did this happen? The answer is simple, different environments. Max prepared and reviewed his presentation through the screen of his computer, not the LCD projector and screen that were actually used to display the presentation. Failing to test the presentation in the environment of the conference room and its equipment helped to sink Max's presentation, which was unable to win over the potential clients during the meeting.
Everyday, people access a plethora of digital products like websites, apps, and software using tons of different devices, supporting various operating systems, screen sizes, and more. It is this diversity that makes environmental testing is vital today.
Environmental testing is the testing of a website, app, or any digital product across multiple devices (environments) to measure the quality and performance of its functions and user experience (UX). In other words, environmental testing is a QA assessment over variety of testing configurations to uncover bugs that are specific to certain devices, network, browsers, operating systems, etc.
Environmental testing makes it possible to find bugs across a number of configurations to ensure the perfect performance of functions and an optimal user experience. Without environmental testing, an app, for example, can offer very different user experiences on various devices as a result of array of screen dimensions, operating systems, and more. Allowing bugs to hinder the UX or performance of functions can lead to users abandoning subpar apps and websites. In fact, 60% of users do actually quit an app or website that offers a poor user experience. Thus, it is essential to execute environmental testing to find and eliminate bugs that can contribute to higher churn rates.
Generally, it is recommended to execute environmental testing as early and often as possible. Given the popularity of Agile, DevOps, Continuous Integration, and Continuous Delivery, testing as early and often is a great strategy for improving the overall development of a website, app, SaaS, etc, while also simplifying bug fixes to remain on budget and on schedule. Nonetheless, a stable environment is needed when conducting any type of QA testing including environmental testing.
In order to perform environmental tests, it is critical establish the objectives of the tests. If the aim of test is to assess the performance of specific functions, it is absolutely necessary to establish detailed testing script with clear steps along with the expected results. Known as a guided or scenario-based test, it not only allows the QA team to determine the performance of the function according to the pre-established goals, but also ensure the tests are executed in a manner that meets the needs of the QA campaign.
For other tests, the aim may be to assess the quality of the user experience. In these cases, it is important create a test that allows the QA team to explore the app, website, or software from the perspective of the end users. Known as an exploratory test, it allows members of the QA team to randomly explore and use the digital product to uncover a variety of bugs that affect graphics, texts, ergonomics, and many more factors that contribute to the overall user experience.
To cover 90% of the risks associated with the diversity of the of smartphone and tablet markets, for example, it is recommended to test using at least 25 devices. It is important regardless the number of testing devices used, to choose devices that reflect the devices used in the target market. In the UK today, Apple holds 51% of the mobile device market while Samsung holds 28% of the market. Meanwhile in France today, Samsung holds 40% of the mobile device market while Apple holds 32% of the market. As a result of these differences, the ratio between Apple and Samsung testing devices should differ when it comes to executing tests for these two markets.
Additionally, using real devices is a sure-fire way to provide greater coverage by ensuring that members of the QA team can test a full array of possible functions as well as truly asses the user experience by using the same devices as end users.