Rendering Rates Using AR and VR

1 Introduction

Virtual reality (VR) and augmented reality (AR) are technologies that are becoming more popular every day. The more the technology advances the more we can learn about it and create better experiences. Virtual reality is when the user is totally decoupled from the real environment and only experiencing the virtual environment without any real aspects contributing. Augmented reality is supplementing real environment with virtual objects. Both technologies are similar but also have different aspects to developing them. In AR there needs to be computer vision to recognize the real objects in the view and then implement the virtual object onto them that looks natural.

For example, with a virtual vase on a table it is important to know where the top of the table is to implement the virtual vase that looks natural. In VR the entire environment needs to be developed and this creates a lot more rendering power needed to create the entire virtual environment. Being able to optimize the rendering rates is important to preserve GPU and CPU. If there is too much power getting allocated to the rendering this can slow everything down and decrease the user experience.

There is a more brute force method that is rendering each eye one at a time called multi pass and rendering the entire environment twice. This takes up a lot more GPU and CPU power which is not ideal. There is single pass which renders a good portion of both eyes at the same time. Using a single pass stereo instancing rendering method both eyes can be rendered together in a more seamless way that reduces the power consumption allowing for more room for other aspects of the experience (Srinivash, 2017).

1.1 Virtual Reality

Virtual reality is taking a virtual environment via a head mounted display (HMD) or a sit in simulator. One aspect of VR is that it needs to completely decouple the user from the real environment. This allows for the user experience to be immersed into the virtual environment. This can be used in the gaming industry or for training certain professions. It can also be used to experience other countries or remote locations like Antarctica. Having the 360 view of the environment allows for the use to explore it with the HMD.

Figure 1 Stereoscopic-television apparatus for individual use (L. Heilig, 1960)

1.2 Augmented Reality

Augmented Reality (AR) is the process of inserting digital objects into the real environment. There would be either a smart phone using the camera to capture the real environment in front of the user and then augmenting objects or characters into the real environment through the smartphone. Another method is AR glasses to have the augmented objects put onto the glasses lens to augment the real environment that is seen by the user.

This was popularized by the mobile game Pokémon Go. It uses augmented characters through the smart phone to interact with them in the game.

Figure 2 Glasses patent for Augmented reality (P. M. Cazalet, 2019)

2 Methods and Materials

2.1 Rendering the Environment

One important aspect to both technologies is rendering the objects with computing power and displaying that to the user either through HMD in VR or glasses in AR. It is important to not have a delay in the system which can cause uncomfortable experience or non-enjoyable experience if it is not being displayed with quick enough speed.

There are a few different methods of providing the information to the device. One is a multipass where the information is being sent to one eye then again to the other eye. This is the brute force method and is very inefficient. There is single pass which saves some steps. The single pass already improves the GPU quite a bit but still has a bit more CPU use. Then there is single pass stereo rendering to render each eye at the same time in a stereo method. This reduces the amount of CPU significantly. It reduces the amount of GPU slightly but even a little improvement can really help in the end (Srinivash, 2017).

Table 1 Rendering Time (in milliseconds) (Srinivash, 2017)

As you can see in table 1 there is a significant increase in CPU time going from Multi pass to single pass and a slight increase going from Single Pass to single pass instancing. And the GPU does not get better from single pass to instancing. But there is an increase in speed on the GPU from multi pass to single pass. And this stays the same through the next process to instancing.

2.2 Software

How the software creates the shapes geometrically and triangulation is also a topic worth touching on. Vertices are used to create all the points through the software to render the shapes and geometry. It is very high-level programming that is going on for this process. The speeds need to be high too so there is not disruption in the visual (McHenry & Bajcsy, 2008).

Knowing where the surface is in the real world would take computer vision along with light detection and ranging (LiDAR) to detect the distance of the real object. The software to pick up the real object is advanced. Along with the software to render the object into a 3D object is also advanced. The software must use code to create vertices to render triangles to render the objects in the real environment for AR.

Figure 3 Virtual bug augmented on a real tree (AR Kit Apple Inc.)

As you can see in Figure 7 the virtual bug is able to crawl along the vertical surface of the tree. In this video on AR Kit’s website, it shows how the bug is avoiding you so when you look and it and go towards it is running away and hiding. It crawls along the edge of the tree very smoothly and feels very real. It is a very good user experience. Being able to input the tree dimensions using LiDAR into the software then rendering the virtual bug it needs to be very fast and efficient (AR Kit Apple Inc.). This is with AR Kit so it is used on a mobile or tablet device not on glasses so there wouldn’t need to be any stereo rendering. As you can imagine there needs to be a very quick response time from real object to virtual object.

Figure 4 Virtual bug laid out in 2D (AR Kit Apple Inc.)

In figure 8 it is the same virtual bug, but it shows how each component is in a 2D map. This is a good representative of how the software works. It has to create each object using vertices and matrices using code then it would map it out in a 2D fashion seen on the left. Then finally orientate these objects into a 3D fashion. It is interesting to point out that the image on the right is actually on a 2D screen, but it is just an optical illusion to make it look 3D and that is an important factor when considering development of virtual 3D objects.

2.3 Sensors

Eye detection in the HMD is also an important part that must be considered in the application. The HMD can keep track of where you are looking within the HMD to know what to display. If the HMD does not know exactly where the user’s eye is then it cannot output, the correct information.

There are other sensors that are important. The location and orientation of the HMD or glasses is important to know the direction the head is facing. The eye location along with head location is required to be able to display the content to the screen or glass.

Figure 5 Position and Orientation of HMD (Kalarat & Koomhin, 2019)

As you can see in figure 3 the Yaw, Roll and Pitch are important inputs that the HMD will need to be able to pick up to render the output of the screen contents. If the direction and position are different and changing or do not match up to what the screen is displaying this could cause a disrupted experience (Kalarat & Koomhin, 2019).

3 Conclusion and Recommendation

There is also a haptic feedback system for VR in the case of the handheld device that could vibrate when something interacts. The rendering rates are important to be efficient to make sure the haptic feedback is on time with what is being rendered visually. Having the stereo instancing will allow for room in the CPU to engage the feedback while not lagging on the visuals. The CPU must be able to output the visuals and the physical feedback. Both systems need to be on time with each other for a better experience. Tracking the location of the HMD is important too. Having sensors in the room to know the location of the user and head device can be good to produce the virtual content. Having the room be able to track the location of the using can help with providing the information to the screen. Render rates can be assisted by this to know where the user is.

Figure 6 HMD with hand controls (Kalarat & Koomhin, 2019)

In figure 4 you can see the HMD used with a hand device to allow for greater level of human machine interaction. Augmented reality can have augmented objects or characters placed on to real surfaces. The surfaces could be a horizontal plane but technologies like AR Kit by Apple allows for augmented objects to be on vertical surfaces too. When rendering the virtual object onto the real surface it is important to not have any latency. If there is a delay in the rendering, then the experience would not be as pleasant. The mismatch of the real surface and the virtual object could create a poor user experience. To make sure there is a smooth and enjoyable user experience there needs to be a quick rendering process of the virtual objects on the real surfaces.

It is also important to be able to render the real objects, the software needs to be able to pick up the real objects using computer vision with good speed as well to be able to augment the 8 virtual objects onto the real objects. So having a quick response time for both features will allow to create a better experience.

Using the stereo instancing rendering system perhaps it is possible to apply this to augmented reality. This would be in the case of a glasses head device instead of a smart phone since on the smart phone there is only one screen. Every millisecond that is saved can greatly enhance the performance which leads to a better experience. If there is less allocation to rendering, there can be more into location tracking of the user.

Using the stereo instancing rendering system perhaps it is possible to apply this to augmented reality. This would be in the case of a glasses head device instead of a smart phone since on the smart phone there is only one screen. Every millisecond that is saved can greatly enhance the performance which leads to a better experience. If there is less allocation to rendering, there can be more into location tracking of the user.

Using VR to teach medical students’ neuroanatomy was tested in a randomize control study. It was found that using VR created better knowledge retention than doing it with the traditional paper-based studying (Ekstrand et al., 2018).

Figure 7 3D rendering of a brain using Python (Hisey, 2021)
Figure 8 Imaging of a slice of a brain using Python (Hisey, 2021)

In figure 7 you can see a 3D rendering of a brain image using Python. I created this rendering from FRMI dataset online (Sarah A, 2020). You can image having this sort of brain rendered in a VR environment and you can rotate the image or perhaps change where it is sliced to see different cross sections. This would create a great user experience when studying brain imaging for certain research points. Figure 8 is a 2D image that is taken from a certain slice of the brain using Python as well. Location tracking in Augmented reality is a feature that can be used to enhance the experience. When a virtual object is augmented onto a real object and the user moves their location in real space then the virtual object could change in shade or colouring depending on the light source. Also, the sound could change it is making depending on where the user is relative to the virtual object or if the virtual object is moving around the real environment. How the sound would reach your ears would change depending on the acoustics of the room so having a variable output from the system would enhance the experience. Being able to know where the light source is, is an important feature in creating realistic shading and shadowing. This can be done with software to pick up where the light is and create a virtual environment of the real space in its logic to predict where the shadows would be with the virtual object in the real space.

We can enhance our performance of AR and VR by using a more efficient way to render and one way is by using stereo instancing to render each eye at the same time instead of one at a time.

4 Bibliography

AR Kit by Apple https://developer.apple.com/videos/play/wwdc2020/10612/

Ekstrand, C., Jamal, A., Nguyen, R., Kudryk, A., Mann, J., & Mendez, I. (2018). Immersive and interactive virtual reality to improve learning and retention of neuroanatomy in medical students: A randomized controlled study. CMAJ Open, 6(1), E103–E109. https://doi.org/10.9778/cmajo.20170110

Kalarat, K., & Koomhin, P. (2019). Real-time Volume Rendering Interaction in Virtual Reality. International Journal of Technology, 10(7), 1307. https://doi.org/10.14716/ijtech.v10i7.3259

McHenry, K., & Bajcsy, P. (2008). An Overview of 3D Data Content, File Formats and Viewers. Technical Report, 22.

Morton L. Heilig (1960) Stereoscopic-television apparatus for individual use, United States Patent Office

Peter Michael Cazalet (2019). Interchangeable eyewear/head-mounted device assembly with quick release mechanism, United States Patent

Rob Srinivash, November 21, 2017. How to maximize AR and VR performance with advanced stereo rendering https://blogs.unity3d.com/2017/11/21/how-to-maximize-ar-and-vrperformance-with-advanced-stereo-rendering/

Sarah Aliko, May 27, 2020. Naturalistic Neuroimaging Database https://openneuro.org/datasets/ds002837/versions/1.0.5

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Cory Hisey

Cory Hisey

I am a Mechanical Engineer graduate and I am currently studying masters of Mechatronics Engineering.