How to Test Smart TV Applications?

Smart TV applications are becoming the new competitive frontier for media companies and content providers. This relatively uncharted territory offers ample possibilities today, but is also rife with challenges – not only in the development phase, but also in quality assurance. 

Inexperienced developers may be tempted to treat smart TV applications just like big-screen mobile apps – and test them as such. This is a flawed approach. Because the differences between mobile apps and smart TV applications are significant, a whole new perspective is needed. 

In this article, we explore the general considerations for testing smart TV applications and discuss the specific idiosyncrasies of smart TV apps that make the testing process different.

The smart TV application testing process

Just like mobile apps, Smart TV applications can be tested using physical devices or emulators. The process should follow a pre-established test plan created using design documentation and product specifications. 

In the initial phase of application development (design and first steps in implementation), the role of the tester is mainly to do a thorough review of the project documentation and, based on it, create test scenarios, plans, DoR and DoD. 

During the tests, any bugs found are reported. Fixes are checked during the tests of the next version of the application. In addition, the main critical paths characteristic for a given type of application (e.g. user login screen and video playback in VOD applications should be tested thoroughly) are part of the regression tests performed with each delivered version of the application, regardless of new functionalities appearing with it.

Design differences between mobile and smart TV apps

Design-wise, one of the most evident differences is the size of the screen and icons. This impacts the layout of the application. Television applications are usually displayed on large-size screens, typically with a 16:9 aspect ratio, which requires the application to properly scale its elements of the interface to make them clear and easy to read. 

On the other hand, mobile applications are dedicated to much smaller devices with different aspect ratios, where the application interface elements are usually displayed in a vertical orientation. This impacts how the application is tested.

User interaction

In mobile apps, the user uses the touchscreen, whereas the remote controller is needed to interact with a smart TV. But because there is no touch interaction – the smart TV app must show the focus (selection square) at all times, making it clear for the user which component on the screen is currently active. 

Source

The testing process should verify that there is no conflict in focus and navigation (for example, when the bottom-most element is selected, another click down is expected to take the user to the next screen below).

If the app supports keyboard input, it should also be tested – e.g. for how it handles keys and what language is used (for special letters).

User interface

One key difference between interacting with mobile apps and smart TV apps is the interface. 

Smart TV screens are wider than mobile devices, which requires a horizontal user interface. This design may entail many changes in the way content is presented to the user and how the user navigates the interface. 

Also, because the distance is bigger (users sit several meters away from the screen and never touch the interface) bigger fonts should be considered when developing the user interface. Quality assurance should involve testing how easy it is to read text on the screen – not only subtitles, but elements of the interface.

You may have noticed that smart TV screens typically use darker interfaces and avoid white elements displayed for a longer time. This results from the need to protect sensitive OLED screens from the destructive burn-in.

Remote control

The remote control is neither user-friendly nor responsive, but enjoys a strong position as the golden standard for controlling TVs. The intrinsic “clunkiness” of the interaction must be considered at the design phase and factored into the testing process.

The essential navigation keys on remote control navigation are: right, left, up and down, and OK. No matter the brand, every TV should have some variation of the standard TV-remote D-pad (see image below). The testing process should confirm the interface does not use the buttons that some brands are not using. For example, the OK button should invariably allow the user to choose items in the interface, and all major manufacturers use it. 

Remote controls come in different shapes and sizes, but the d-pad is the industry staple and remains unchanged across all brands (source).

The UI layout of the smart TV application is very important in the testing process and app navigation plays a very important role in creating the test generation strategy. Understanding the layout could lead to an efficient test generator and runner. For example, Smart TV apps typically follow one of three main patterns. 

Layout (b) is mostly used in smart TV applications as it minimizes how often views are changed (source).

The limitations imposed by the remote control should be reflected in the navigation in the smart TV app. For example, in the case of non-adjacent views, more than one step is needed to move from one view to another. This is not convenient when using a remote control and should be minimized in the design of the app. 

Unique TV functionalities

Even the most skilled Quality Assurance Engineers have no way of knowing the specific features of all TV models. TVs particularly lack standardization in terms of their “smart” functionalities (compared to mobile devices). This means that two different models (even made by the same manufacturer) may not have the same features or offer the same performance.

For example, specific Samsung TVs with Tizen offer multitasking (apps can run in the background), but you need testing to confirm your smart TV app doesn’t take advantage of a feature which is absent in 90% of TVs. By comparison, mobile devices don’t have this level of fragmentation, which makes testing much easier, and makes it possible to make certain assumptions. 

Important considerations for testing smart TV applications?

Testing Smart TV apps is a complex process due to the sheer number of vendors, models, and operating systems available on the market. It is essential for app developers to get their application tested for the specific model, platform and operating system. Here are a couple of good practices for testing smart TV applications:

• Use manual testing – Emulators come in handy for smart TV app testing as it is not possible to test on all versions of smart TVs as they are pretty expensive. However, there are no emulators for specific platforms like ROKU, and the emulators for other platforms may not support testing for all functionalities.
• Test different video formats – to make things even more complicated, testing should also cover the support for specific video formats across many different devices. This involves testing content across various brands, year and model groups, encryption solutions, playback or player features like subtitles and internet connectivity specifications.
• Test for bad network – Most smart TV apps are designed to connect to the internet whereas this is not the case for the mobile apps. Testers tend to overlook scenarios where network connectivity is bad – the app freezes or closes abruptly. Testers need to test these use cases so that the right error messages are displayed in case of low network connectivity. 
• Test for different resolutions – an app may work fine on one smart TV with a particular resolution and freeze on another as smart TVs do not support pixel CSS. 
• Test usability – Navigation in smart TV apps should be simple and easy to understand. 
• Visual rendering should be of high quality, sharp (alignment and spacing should be taken into consideration) and should adjust as per the screen size available and resolution should be optimal. 
• Loading and buffering should be indicated by the loading indicator
• Search should allow enriched browsing and content discovery.

Conclusion

Content providers, IT firms and TV manufacturers are increasing investments in smart TV applications. For streaming platforms, how the TV app is designed and how well it performs can make or break its success. Poorly performing apps fail to engage users, don’t encourage subscriptions and have a negative effect on the brand’s image. This only shows that testing TV apps is a big deal and shouldn’t be overlooked – exhaustive test cases should be developed for all smart TV platforms, and complexities should be defined.

If you’re in the process of or considering developing a smart TV app, shoot us an email – we’re always willing to talk! Our experts will offer a free consultation of your project.