Virtual and Augmented Reality are no longer imaginations out of science fiction movies. These ‘futuristic’ immersive experience technologies are actually getting better in aspects like display quality, compute power, and more so tracking capabilities. In fact, they are already changing the way we interact with immersive content.
You must have noticed that buzz around the likes of every new version of the likes off Meta Display and Apple Display headsets.
The technologies are also shifting away from traditional interactive ways like typing to techniques like hand-tracking, virtual keyboards and even eye-tracking to deliver amazing immersive experiences. In this article we will delve into how VR and AR works including how typing looks like in the world of AR & VR and what the future holds in store for us.
What are VR and AR Technologies?
If you haven’t heard about them before, then you’re probably wondering what VR and AR are in the first place and how they work.
Well, VR simply stands for Virtual Reality. It is a technology that creates what’s pretty much a complete 3D digital environment. So, in the place of input devices like traditional keyboards, VR makes use of virtual keyboards in the virtual world.
In this case, hand controllers are used for pointing and clicking on virtual keys in most cases but in some, voice input can be used.
Other than this, gesture-based typing is also used. This is where hand movements are tracked and interpreted as keystrokes then fed back to the device as interactions.
AR on the other hand stands for Augmented Reality. This tech essentially inculcates digital elements into the physical world including the likes of floating virtual keyboards or the virtual keyboards that are being projected onto real surfaces. The technology often uses hand gestures though voice input can also be used.
The goal is to reduce distractions to you even though you pretty much still stay in touch with the physical world.
How Typing Works in VR and AR
When it comes to VR, the user is completelyimmersed in the digital world. But how exactly is this achieved?
Now, the VR software generates a 3D virtual environment with a virtual keyboard. Headsets and controllers are used to track the user’s movements to make sure that the environment responds well.
You type using the virtual keyboard and the texts appear in the virtual environment. In others, voice commands, hand-gestures are interpreted as keystrokes.
As we’ve mentioned earlier, AR adds some virtual tools in the physical environment. So, generally speaking your hands and your surrounding environment are captured by the AR device’s camera. The AR software identifies your hands and maps the surrounding environment and analyzes the image.
Here your hand movements are tracked and then a virtual keyboard is projected in front of you or on a solid surface for instance on a surface like a table. As you type the virtual keys, the software inputs this. Alternatively, you could move your fingers and the AR software will interpret this movement as keystrokes.
Key VR/AR Typing Technologies
Now that you know a thing or two about how VR and AR work, let us now look at some of the examples of the groundbreaking technologies that are making VR and AR typing possible:
Hand-Tracking & Finger Typing
Now, current VR & AR technologies are already making use of advanced hand-tracking systems in the virtual environment.A good example is the Quest 3 VR headset device by Meta (the parent company of Facebook) which allows you to type using natural finger movements either via typing a virtual keyboard or swiping in pretty much the same way you would in a smartphone.
The Quest 3 actually supports hand tracking (for some apps). This means that you can use your hands directly to interact with virtual objects, including an on-screen keyboard. Yes, you can simply use your fingers to “press” the keys on that virtual keyboard.
You could point at them and tap your fingers in the air which certainly makes the typing experience feel more ‘traditional’ or rather, natural and intuitive.
Voice-to-text typing
Current AR/VR technology is also tapping into voice-to-text typing. A great example is Meta’s voice assistant which is now integrated in Meta Quest devices. This feature allows users to interact with the virtual environment through speech. For example, you could say something like Open browser and the voice recognition translates the command then the action is executed.
You can then tell it to visit a given website on the browser. This voice dictation can be used to help you type faster with VR/AR headsets.
This technology is however pegged on the accuracy of the speech recognition system, plus it’s best used in a quiet environment.
Challenges in VR/AR typing
Despite the major advancements made, VR/AR still faces some challenges and limitations For one, since the whole VR/AR typing involves being in the digital environment, it’s not easy to adapt your brain to typing in such a virtual environment, especially if you’re used to getting that tactile feedback with the physical keyboard.
It’s also worth noting that most of these technologies are still at the infancy stage in terms of development. For example, most of them are yet to track both hand and finger movements smoothly.
In the case of speech commands, then the speech-to-text systems can have errors like say, the speech can be misinterpreted plus there are pronunciation barriers that are a limitation to some user demographics.
Also, since hand-tracking is also involved, virtual keyboards tend to be slower given that lags tend to happen.
What Does the Future Hold for VR/AR Typing?
All said and done, the future of VR and AR looks quite promising. There are many studies and innovative work on how to address the current VR/AR challenges.
For example, there is already work to narrow the gap between real-world and virtual typing experiences. Many of the AR/VR manufacturers have been evolving and creating wearable devices that emulate the feel of actual keyboard keys with every new product iteration.
Research work is also underway to improve surface typing detection or rather, that ability to type on any physical surface. AI is also taking over and we expect this to get even better in the coming years.
These headsets will certainly get better at predicting the user’s intentions and even correcting errors in real time to improve typing experiences.
Who knows, maybe soon we’ll be in a position to type exactly what we are thinking as we use AR and VR headsets!
Wrapping up
In a nutshell, VR and AR have already changed how we interact with content including how we type in the virtual world. Technologies that looked futuristic just a few years ago like hand-tracking, virtual keyboards and even voice inputs are now very much a reality. This is just the beginning, though. We can expect that VR and AR tech will get better in future and so will typing in that virtual world!