As over-the-top (OTT) continues to establish itself as a key growth area within the media industry, new players are hurrying to join an already crowded market. In this increasingly competitive landscape, providers ought to ask themselves how to make sure that their service stands out from the crowd. Innovation is important, of course, but it cannot come at the expense of reliability. Consumers have too many competing services available and the abundance of choice empowers them to be much more selective than during the early years of OTT. This is also true for emerging markets, where numerous providers are already setup and ready for the day when the required bandwidth and infrastructure are available for consumers to subscribe to OTT and video-on-demand (VOD) services.
The OTT gold rush has led many providers to dive into getting a product out as quickly as possible, without giving much thought to the viewer. This can have disastrous consequences for the overall quality of user experience (UX) and leaves many consumers frustrated with the service. Challenging navigation, a clunky user interface (UI) and constant stalling and rebuffering are just some of the issues that often are overlooked in the rush to get an offering to market. This results in a high level of churn before the service has gained the engagement it needs for long-term success.
Testing Your Patience
For those that doubt the importance of quality control and testing, this 2020 report from Capgemini and Sogeti shows how vital it can be. According to the study, 85% of executive management teams across the media industry believe that quality of service contributes to business growth and improved outcomes as well as protecting corporate and brand image.
Despite this, as with any commercial project planning, a balance needs to be found between releasing a pixel-perfect service and keeping stakeholders happy. With manual testing, the cut-off point is reached quite early in the development process as the cycles are extremely time consuming and can be prohibitively expensive – especially when budget needs to be reserved for ongoing manual monitoring post-launch.
With an automated approach, thorough testing can be achieved in a fraction of the time. Aside from the huge benefits of increased speed-to-market and the associated cost-effectiveness of reducing development time, automated testing is also reassuringly reliable. Test automation enables teams to take a rigorous approach to quality control by emulating multiple combinations of user behavior. The result is a much more dependable service, a better early adopter user experience, and a solid base on which to build future releases and updates.
Four Areas of Focus
Automation certainly represents a great foundation to improve the quality and efficiency of a service. By reducing the dependency on costly and time-consuming manual testing, services stand a much better chance of success post-launch. With this in mind, there are four important areas to consider when planning automated testing.
- User Interface (UI) Testing –To validate important user interactions, the UI testing must simulate how different users might engage with the platform. The testing framework observes changes in the UI, to ensure visual functionality is working correctly. This testing rapidly identifies regression issues and validates important user interactions. Comprehensive UI testing should always include the automation of regression tests including business-critical functionalities.
- API-Driven Testing – Although invisible to the consumer, back-end reliability is crucial to their viewing experience. This testing framework bypasses the user interface to identify changes to service layer integration points and responses. It validates the overall behavior of the service under test conditions. Fast-track API automation will verify key integration points and the health of back-end systems.
- Unit and Component Testing – The code is the foundation on which the entire platform sits, and issues in this area can be notoriously difficult to rectify post-launch. This stage of automated testing verifies the integrity of the crucial underlying code.
- Manufacturer Compliance – This area of testing ensures that the app development adheres to guidelines from OEMs. A thorough automated testing approach should target high-traffic platforms/devices and expand to cover multi-platform portfolios.
Automation in Action
As competition in the OTT sector increases, so has the demand for automated testing. On the surface, any automated approach can seem like a magic solution that can only result in a flawless product release. However, this thought process underestimates the complexity of automation and is an extremely risky way to approach testing.
Automation is a robust and effective approach but as with most development processes, the results are dependent on the expertise you put in. The complexity involved when deploying an effective automated setup can challenge even the most experienced teams. A recent report found that 48% of companies find it hard to recruit people with the right skills needed for automation projects.
In their hurry to roll out a solution, providers can make the mistake of trying to implement an “off-the-shelf” testing approach. This can leave holes in the testing process, such as limited tooling or poor automation rates, which cause huge issues after the service has been rolled out. A customized approach ensures comprehensive test coverage for a provider’s specific requirements. Early detection of issues is crucial; the earlier a bug is identified, the more cost-effective it is to resolve, and this lowers the risk of it affecting end-users.
The Video Ecosystem
A classic pitfall when bringing an OTT product to market is that the service might not be fully compatible with one or more of the wide selection of devices available. Viewers may find that the UX is perfect on their main screen but has an awkward or buggy interface on their tablet. In the early stages of launch this can spell disaster for subscription retention. For established services releasing new updates, such problems risk jeopardizing consumer loyalty and satisfaction. The fact that viewers may engage with the service across multiple platforms cannot be overlooked. Providers need to ensure that their consumers can transfer a viewing experience seamlessly from one device to another across multiple platforms.
To maintain consistency across multiple platforms and big-screen devices, OTT providers must not only look into mobile/web automation but also move towards real device-based automation on big screens. With a customized approach, it’s possible to factor in the individual nuances within the service under development. Experienced automation teams will consider everything from the engagement model to the service lifecycle and the projected number of deployments to market.
To deliver this effectively, teams need to have broad video ecosystem expertise. Automation of playback on mobile devices and big screens is key to ensuring better quality of services and flawless playback. This helps to keep end-users happy. Products with a complex back-end involving multiple third-party integrations will see particularly valuable results from robust automated testing.
Maximizing the End-User Experience
Leveraging automation to avoid time-consuming and costly manual regression tests over the product lifecycle naturally improves overall efficiency during both release and subsequent deployment stages.
There are multiple providers that would see significant benefit from automating the testing process, and these benefits are not limited to new services. With comprehensive testing, established services can confidently explore innovations in UI and UX for new releases.
Only with a holistic, video-centric approach can a provider effectively manage its initial investment against subsequent release budgets while ensuring an engaging and scalable offering over the long term. An experienced testing team will be able to work alongside providers to understand their product vision and automation ambition levels. A customized solution will deliver the right approach as well as the most suitable automation tool.
Configuring a bespoke automation environment (including test scripts, reporting and ongoing management) means that testing will fit within the provider’s project plans rather than dictating them. Ultimately, automated testing ensures that video providers can maintain quality of service regardless of the device on which the service is being consumed. This in turn will keep subscribers happy and help minimize the risk of churn.
Poornima Urs is director of QA engineering at Accedo.