Why We Play

video games | culture | technology

(607) 279-9770

Revealing VR Controls

With the Oculus Rift ostensibly just a year away, it strikes me that my keyboard is a bit antique.  To be sure, it's a USB-connected, mechanical keyboard with fancy switches and a satisfying action, but I suspect that it's about to be rendered invisible.  Let me explain.

The Rift and other virtual reality headsets immerse you in a 3D world which completely fills your vision to the periphery (unlike the CastAR, an augmented reality headset).  The Rift does this by using a small, high resolution display and optics to present each eye with a different view, exploiting our stereo vision to produce images with depth.  I'm eagerly looking forward to the day I can pull one over my eyes and disappear into a game like DayZ.

While there is a great amount of excitement and support for the Rift among game developers, Oculus doesn't think that consoles are open enough to support the device, and Sony and Microsoft are apparently developing their own VR headsets.  So for now, the Rift is firmly a PC-connected device.  And one of the hallmarks of PC gaming is its traditional control scheme: the keyboard and mouse.

It remains to be seen what role the mouse will play in Rift-supported games.  The device tracks head motion through gyroscopes, accelerometers, and magnetometers.  Instead of gliding a mouse to look around a game, you simply turn your head.  Some players may choose to use the head tracking to aim in shooters , but other gamers will want aiming that's independent of head motion.  This can be accomplished with a mouse, as it has been for so long, but tastes change, and other gadgets may take the place of our trusty pointing devices.

We've already got motion controllers like the Razer Hydra, the Leap Motion Controller, and the PC-compatible version of the Microsoft Kinect.  Some sort of device that reads your hand waggles would seem a natural match for head-controlled game viewing.  Such a pairing could strengthen immersion and assist spatial orientation.  If your in-game head and arms accurately track your real ones, you feel like you're inhabiting that virtual body more completely.

Where does the keyboard fit into this scheme?  PC gamers have long used them and seen their multitude of keys as a boon, especially in more complex games.  If virtual reality is coming first to PCs, developers and gamers need to think about how they want to use keyboards for VR games.  All those keys present a problem.  If you're like me, your fingers will spend much of their time on the WASD home cluster and surrounding keys.  These keys are easy to manipulate without glancing at them, but if you're playing something like a real-time strategy title, it's not easy to accurately choose a key toward the right side of the keyboard by touch.  And if you're playing a realistic flight sim with a keyboard and need to lower your flaps, forget it.

How might we make our keyboards compatible with VR headsets?  We could make the lower quadrants of our headsets partially transparent, so we could look down and see our fingers and keys.  But this would break immersion, and you don't need to be able to see your keyboard the majority of the time.  You'd always have the real world dancing at the edge of your vision.  No thanks.

I propose a virtual keyboard overlay.  It might use a keyboard with capacitive sensors built into the keys.  There are already keyboards with capacitive switches, but these sensors would serve a different function: they would detect your fingers touching or hovering over the keys.  Developers would add code into their games and applications that allowed a virtual keyboard (with virtual fingers) to appear as a HUD (heads-up display) element laid over the graphics.  Gamers would be able to choose the opacity of the keyboard and determine when it appears.  Because a VR headset already has motion tracking, simply tilting the head to look down might cause the overlay to fade in, allowing simple and intuitive interaction.

In some ways, these virtual keyboards would even outdo their physical counterparts.  If you preferred, you'd be able to look right through your "fingers" and see the labels on the keys.  These labels could be customized with the names of commands, so you might see "select all" instead of the letter "A".  There are many possibilities.

Eventually, you might just hover your hand in the air and press virtual keys instead of plastic ones.  Or climb into a Virtuix Omni and run around with your whole body.  My guess is that as the options increase, gamers will pick the combination of input and output devices that work best for them.  And I'm all for that.