With the development of technology and the use of applications, a brand has the opportunity to be almost 24/7 in the hands of its users. And they are demanding. In a market that is guided by users’ expectations and where quality and performance are determining factors, creating a seamless user experience is a key to success. When facing a bug, a crash, or having a rather unsatisfying experience users will leave the app and chances are they will also delete it.
By using app analytics tools and setting metrics, developers, testers and marketing teams can collect and analyze data concerning the way the app operates and the behaviors of its users. This allows them to carry out corrective actions, modifications and improvements deemed necessary in order to bring an ever-increasing level of satisfaction to users.
A 2016 study by Gartner shows that organisations make too little use of app analytics. This is especially true for their B2E (business to employee) and B2B apps. A huge amount of information is produced by the use of apps and firms are still finding it difficult to establish usage patterns and to leverage opportunities for improvement. By 2022, only 50% of all apps on the market will use analytics tools to improve their users’ experiences.*
In order to be able to analyze the data collected throughout an app’s life cycle it is crucial to set metrics related to operational analytics, to behavioral analytics, as well as to the performance of the testing phase itself.
Metrics or KPIs enable an organization to measure results according to parameters that are considered relevant. Each organization will thus have to decide on the results depending on their project portfolio, objectives and workflows.
1. Setting objectives
Each metric has to bring an answer to a question. Organizations thus have to determine which answers they are looking for. This can be related to the performance of the app itself, to user engagement or to business results.
2. Setting metrics
In order to decide on the right metrics, they have to be compared to one another first. This will help point out which ones best correspond to the objectives that have been determined. A combination of multiple metrics can sometimes be the most pertinent option.
Operational analytics give insights on the availability and performance of mobile apps, relating to the device, the network, the server and other technological factors.This type of analytics is essential in order to capture and fix unexpected behaviors of the app (such as crashes, bugs, errors or latency), which can lead to user frustration.
An organization could for example look at metrics such as processor usage (CPU), APIs performance, response time, crash reports, etc.
Behavioral analytics focus on collecting valuable data on how users interact and engage with the app. Monitoring the real usage of an app in the long run helps towards setting benchmarks and comparing « before/after » periods of time (pre-release, post-release). It also stresses the friction points within the app which could be hiding bugs, or which could show an evolution in behaviors or even an unsuitable UX scenario.
Relevant metrics could be download rate, open rate following download, active users per month or per day, length of sessions, frequency of sessions, time spent per session, transaction rate, frequency of transactions, revenue, but also the journey of the users or the app rating on the App Stores.
Along with the use of metrics, an organization can adopt a continuous testing culture. Whether done internally or conducted by an external provider this will enable an organization to help its teams produce qualitative apps.
Specific metrics can be put in place to monitor testing phases. This will allow for a better understanding of the critical points at the end of a testing phase and to better plan the next testing campaigns, knowing beforehand the functionalities that require the most attention.
Those metrics could focus on:
- the percentage of critical defaults and of undetected defaults before the release enabling the team to investigate, learn from those bugs and to work on ways to avoid reoccurrence during the following testing phases.
- time for testing
- the volume of defects per macro functionality
- the typology of defects (functional/graphical/technical)
- the performance rate of corrections (number of defects reopened VS number of defects fixed).
- the time required for an item to go from « in testing » to the final status, in order to measure for example the complexity of a task.
- the percentage of dismissed defects, meaning the defects found in the app but not accepted as such by the developer. The organisation can better understand if project managers and testers are on the same page regarding the purpose of the features and of the app itself.
- the test coverage, that is, the percentage of prerequisites that have been covered by testing.
- the ratio of defects found during the testing phase compared to the number of defects found after the correction phase.
- and much more.
Applications are increasingly becoming the favored digital interface for brands. Simultaneously, time spent on apps is ever rising. Organizations thus have to constantly strive to improve their services and users’ experiences.
Combining the use of metrics that help analyzing areas of improvements with testing phases throughout an app lifecycle will allow organizations to ensure that their users have the positive experience that they are looking for and to optimize their business results.
* Gartner, Market Guide for Mobile App Analytics, Décembre 2016