A good quality VR experience requires high consistent frame rate, low latency between head motion and display output, and frames that have to be pre-distorted to offset the effect of looking at the display through an optical lens.
Virtual reality (VR) is described as a 3D, computer-generated environment that a person can explore and easily interact with. It is an artificial environment created and presented to the user in such a way that the user believes and accepts it as a real environment.
The technology utilises VR headsets or multi-projected environments to generate realistic sounds, images and other sensations that can simulate a person’s actual physical presence within a virtual or imaginary setting. Being the future of technology, with escalation in the popularity of VR applications, demand for VR testing is also increasing.
VR and augmented reality (AR) often depend on specialised hardware. Proper functioning of these products can be tested by using the devices listed in the requirements, although VR and AR apps need more extensive and in-depth testing because these are more sophisticated software.
Objective and purpose of VR testing
Visual details and display performance are the biggest aspects that are monitored while testing a VR headset. Not only does the resolution shown to each eye matter, but refresh rate, pixel structure and viewing angle can mean the difference between an immersive VR gaming experience or tunnel vision and motion sickness. The purpose of testing is to evaluate the overall impact of the environment on the device and how all this affects the user in the virtual environment.
Design and ergonomics are also important factors considered during testing. VR can affect different people in different ways, but a well-built design and an easy-to-adjust headband can make the experience enjoyable.
Hardware and software testing in VR applications is a challenge that needs innovative thinking, creative solutions, and a strong combination of technical skills and visual acuity. Firms need innovative approaches for manual and test automation to deliver quality consumer products, service and content in this space.
Diagnosing performance issues
In VR, if you are rendering frames at less than 60fps, some frames appear to render at 60fps and some at 30fps. This is because frames that miss Vsync interval have to wait for the next sync point. To optimise VR user experience, app rendering should be kept at 60fps or above. This helps maintain VR immersiveness and prevent VR-motion discomfort.
Rigorous compatibility testing aids in measuring performance of the app when accessed by different devices and platforms. It also helps identify dangerous non-functional issues or bugs, such as device overheating.
Usability is a quality attribute that indicates ease of use of a certain application. VR is an artificial environment created by the software. This way of perceiving the environment is completely immersive and supported by two essential physical devices, headsets and computer/console/cellphone, which create the virtual environment.
Pre-production testing checks whether the use-case is aligned with preferences and needs of the target audience.
A good quality VR experience requires high consistent frame rate, low latency between head motion and display output, and frames that have to be pre-distorted to offset the effect of looking at the display through an optical lens. To ensure a good VR experience, the tool captures and tracks the following key performance metrics:
Frame time
This is the time it takes to render each frame. The lower the frame time, the greater the possibility that the app will maintain a frame rate of 90fps.
Dropped frames
A frame drop occurs whenever the frame rendered by the VR game arrives too late for the headset to display. This causes the game to stutter and increases perceived latency, which can result in discomfort.
Warp misses
This occurs whenever a new or re-projected frame is not created in the current refresh interval. The user experiences this miss as a significant stutter.
Synthesised frames
If FCAT VR detects a lot of asynchronous space-warp frames, the system struggles to keep up with demands of the game. A synthesised frame is better than a dropped frame but is not as good as a rendered one.
Tools for VR testing
There are many tools available in the marketplace to test readiness of the PC for VR, as follows:
OptoFidelity BUDDY
It is a comprehensive solution for VR, AR, mixed reality (MR) and head-mounted display (HMD) performance testing and calibration in both R&D and manufacturing lines. It comes with an integrated vision module and the desired number of degrees of freedom, which ensures the best HMD performance. It also measures audio-to-video synchronisation and motion-to-audio-latency for spatial audio.
FCAT VR
This helps measure performance of VR PC games and prevents stuttery unresponsive gameplay, which can lead to eye strain and discomfort. An FCAT VR analyser analyses performance and quality of the experience in a graphical user interface.
The common evaluation method is to experience VR and see how it feels in motion. As stutter and latency can degrade the experience to the point of motion sickness, these are critical to be measured while VR testing. FCAT VR provides comprehensive performance measurement for frame time and stutters on the VR headset without the need for special external capture hardware.
360° eyeTracker software solution. BluFocus has developed a unique VR positional tracking tool that allows analysing a user’s behaviour within a 360-degree environment. It tracks position and orientation of the user’s head movement while he or she is in a VR experience to extrapolate the field-of-view (FOV). This allows the user to see if visual, haptic and audible cues are directing his or her attention as desired by the storyteller.
Tools for VR app testing
The primary step in diagnosing performance problems is determining whether the app is CPU- or GPU-bound. VR app testing depends on specialised hardware. To make sure that VR hardware and app function properly, these must be tested on different devices and against all requirements.
VR headset manufacturers have created a series of free tools to help potential customers make an informed choice. The tools measure frame time on a device and identify low-performance spikes. Some such tools are Oculus Rift Compatibility Tool, HTC VIVE Test Tool, SteamVR Performance Test and Microsoft Mixed Reality PC Check. One can simply download the tool and check the computer and display the result, indicating each component needed to play VR games.
Some other tools available in the market for VR testing are:
- Unity Profiler and Unreal Profiler for their respective engines
- Performance HUD for real-time monitoring of performance on device
- GAPID for monitoring specific graphics calls
- Android Studio’s Systrace for detailed performance analysis
