There's an inevitable moment when you're in a virtual reality (VR) experience and you look down. Maybe you wish you hadn't, because it's at that moment that you don't see your hands.
It's then you remember that you're not really in outer space, or driving a Formula 1 race car. It's often unexpected and usually disappointing. You can sense your hands holding the controller in your lap, yet when you look down or hold them up, they don't manifest before your eyes. You're taken out of the experience, even if just a little bit.
Manus VR hopes to solve this problem, and do so in a more immersive way than other virtual reality controller solutions for headsets like Oculus Rift, PlayStation VR and HTC Vive Pre.
Instead of holding a controller, Manus uses gloves to not only bring your hands, but also your fingers into the virtual world.
"What we often see is that as soon as somebody puts on the [VR] headset, the first thing they try is to grab stuff, even though their hands aren't there," Jules Blok, lead programmer for Manus, told techradar at GDC 2016 last week. "Even with the motion controllers, we still see people putting them away to try to grab something with their hands. So, clearly, there was a need to track this."
The Netherlands-based team decided to create a solution that uses existing positional tracking of VR controllers mixed with sensors in its gloves to measure hand pose and finger movements. Dual sensors in the fingers help track grips, bends and points, letting users turn their motions into functions.
Finger sensors may sound like the obvious and easy way to bring your digits into the digital world, but Blok says it's far from it.
"A lot of what we've heard from other companies who've tried to use flex sensors is they've had trouble calibrating them," he said. "We've put a lot of research into having a good calibration profile that will work for a wide range of different hands."
The result, as I'll describe below, is wondrous. This was just a small demo at a convention, and there are improvements to be had, but Manus is on the path to perhaps one day solving one of VR's most pressing problems.
All hands
The gloves Manus let the public try at GDC are its engineering sample. The development kit, shipping in Q3 2016 for $250, look more like bike riding gloves, but those were housed behind a glass case.
When I sat down to try Manus' demo, the first thing Blok did was strap two HTC Vive Pre controllers to my forearms. I'd certainly noticed them on other users beforehand, and Blok explained this is a temporary solution for positional tracking. Eventually, the team wants to build custom bracelets for each headset that will integrate with the specific systems.
Next Blok helped me slip on the gloves, which had a polyester feel. I could tell there were sensors sewn in, and a small rumble pack offered some weight, but they were soft and offered decent dexterity.
Finally came the headset and headphones, and suddenly I was in a VR dollhouse where I had to help an orphan find her way home by solving a series of puzzles.
It wasn't immediately obvious what I was supposed to do (there were no instructions or visual cues), so for a minute or so I simply played around with my hands for the very first time in virtual reality.
It was delightful. The hands I saw - a pair of bluish-purple geometric mitts that are see-through to the skeleton, for some reason - responded to whatever I was doing in the real world. I could make a fist, point with one finger, and move my hands above and around me, and the virtual hands mimicked my gestures.
The response time was generally good, though there were occasional lags and glitches. The hands also disappeared if the glove slipped too far from the Vive controller, so I had to pull them on a little tighter or risk my hands vanishing before my eyes.
When I "touched" an object - there were knobs, levers and piano keys I could press down on - a slight rumble and the corresponding movement of the object wasn't quite enough to make me feel like I was interacting with a tangible object, but it was enough for a game that's set in a fantasy realm and still a demo. Just the fact that I could use my hands and fingers to interact with the game at all was impressive enough for the moment.
Eventually, Blok gently pulled up one of my headphones and told me what I needed to do, so it was time to put my hands - all of them - to work. First, I had to grab floating fireflies, push down on a lever (to open a corresponding lantern), and put the fireflies one-by-one into the wire frames to light a path for the orphan.
To perform the movements with my real hands and watch the virtual objects respond as if I was actually grabbing, pressing and moving them was supremely satisfying.
Next, I had to clear away and reassemble a series of broken rock pieces into a staircase for the orphan to climb. This was more buggy than the first puzzle as the pieces had to be put in the right position to hold, and if I moved them too quickly or my grip didn't lock on at the right spot, the pieces would tumble to the ground.
But it was challenging in a good way to grab a rock and move it by clenching my fingers and extending my arm to where it needed to go. I could even subtly manipulate the pieces into position with small finger movements, as if I was performing a precise task in the real world.
Then came a simple errand of pushing down plugs with my index finger to let the orphan through a hallway. My final challenge was to imitate three-key tunes on a virtual piano. I won't say it felt like I was playing an instrument, but again there was that satisfaction in pointing and pressing in real life and watching an action in the game result from it.
After a cheery goodbye, the orphan skipped her way home and my demo was over. I would have liked to play it again.
What struck me most about the experience was how much a part of the Pillow's Willow world I felt because I used my hands. Instead of being a passive observer to the action, I was the action. Whatever movement I performed had a direct impact in the orphan's virtual world, and that's something I've never experienced so acutely before.
While you can pick up objects using the Oculus Touch controllers for Rift, you're not really opening and closing your fingers to do so. Instead, you press and hold a trigger button. So to make a fist around a rock and see it move in real time was a huge departure from what I've experienced with VR previously.
It's not all fun and games
While my hands helped draw me into the purple fantasy realm of Pillow's Willow, Manus has a long way to go before we're all wearing gloves and picking up objects in VR.
In order for it to achieve the device agnosticism it wants and build custom bracelets that work for each headset, Blok said VR companies have to open up their positional tracking systems. Oculus hasn't done that yet, though the company is working with Razer's OSVR. And while its dev kit will ship with HTC Vive Pre support, Manus is still in talks with the firm to gain access to its hardware.
It's an obstacle, but Manus VR co-founder Bob Vlemmix is optimistic headset makers will come around based on the positive response the gloves have garnered.
"We even had [Oculus founder] Palmer Luckey try [the gloves] out," Vlemmix said. "He said it was great, and really cool."
It doesn't end with hardware support, however. In order for users to have the best experience, Blok said games will have to be built specifically for hand tracking. This, again, involves open SDKs, which aren't available across the board yet.
What's more, with developers just beginning to crack the surface of visual VR games, will they really want to focus on developing titles that incorporate hands, too?
"That's one of the things I'm always thinking about," Vlemmix said. "It does add to the level of immersion ... We're getting a lot of requests from developers. They really want it, so I don't think that's an issue."
Manus wants to deliver its gloves to consumers in 2017. It plans to add improvements, like thumb and finger spread tracking, for an even more accurate read of hand movements. It's also working on improving the physics so that you can get a sense of weight and shape from the virtual objects you interact with.
There are clearly challenges to turning Manus gloves into a viable solution that will satisfy VR's hand needs, though from my short time with them on, I can say that going back to an eyes-only experience will be incredibly hard.
Despite being a fairly basic demo, this was the most interactive virtual reality experience I've ever had, which made it the most immersive, too. I wasn't just playing a VR game: Manus VR gloves let me become part of the game.
Are the gloves perfect? Not by a long shot. Could this be the future of virtual reality? It absolutely feels like.
- Our updated hands on review of PlayStation VR
