Leap Motion & ZED Camera Development
We were lucky enough to do some serious experimenting and getting down and dirty with leap motion and ZED camera on a new project that will be released in June.
To give you a brief rundown on what these two technologies can bring to the projects the leap motion allows the player to use their hands and interact in a virtual environment. We also want to bring in some mixed reality components into our virtual environment so we will give stereo labs ZED camera a test to see what it can do.
Findings on the ZED Camera
We thought the ZED Camera would provide a solid solution allowing us to bring in real objects, scan them and then project them back into the virtual environment.
The ZED uses binocular vision and high-resolution sensors, to tell how far objects are around you both indoors and outdoors. The ZED, is also meant to a 3D map your environment for real-time obstacle avoidance, visual effects or world-scale AR.
Unfortunately the way the ZED analyses the environment is not up to a professional standard. The two stereo cameras can’t pick up definition or depth in low light and the results in lit environments still don’t hold up. It also requires a lot of USB bandwidth and can be very unpredictable especially when also running a vive/leap/etc over USB at the same time.
We were pretty disappointed in this product. It definitely needs an addition of infra red technology to improve its accuracy and depth sensing. Also the positional tracking to get positioning and orientation drops out quite frequently so the camera needs to rescan environments each time
Overall our testing for a professional use of the ZED camera could not lead us in many directions. The only use we could see was keeping it completely still on a tripod and scan the environment that way. It could potentially be used for mixed reality filming taking out the background but lighting would need to be absolutely perfect for that to work well.
Infra red is really important in mixed reality production, the new Intel real sense cameras and the Kinect devices have many more use cases than the ZED camera. We will be waiting for major improvements both on the technology side and software side before recommending the ZED camera products.
Findings on the Leap Motion
I had heard some very mixed reviews about the Leap Motion before we decided to give it a solid test. These all centred around it being a very buggy device, hard to program with, and not much reference material available.
We were all very surprised during our R&D stage that this product actually does what it says it can do. It held up really well with our testing and everyone is really impressed. Apart from a few issues with some hand gestures where your fingers are occluded and not picking up certain poses, it is definitely a tool that will see its way into more VR and AR experiences as the headsets incorporate this technology.
In our project we are picking up objects moving them around, plugging them in, all using your real hands. Also have gestures for yes and no commands, typing, menus and UI all hooked up to the players hands.
We tested picking up objects by pinching them but this felt very unnatural in the leap motion. You can’t feel the real object so when you are pinching thin air and seeing the object in VR it is a weird sensation. The player needs that haptic feedback so we came up with a method where the player can pick up objects without the need to grab them.
After all this testing we felt the best way to feel an object is actually not to feel it at all. By disconnecting it from a grab motion the brain can more easily adjust to the new action. Also the fact that the object is not touching your hand means you don’t need that tactile feedback.
Also the tracking with the leap motion can occasional shift and drop out which causes the object to get funky in the environment. So figuring out new ways to pick up and interact with objects in the scene is really important.
All in all we feel the leap motion and using your real hands in the virtual environment is definitely the future of virtual reality. Adding in real objects that you can touch and feel adds a great sense of immersion and we will look to incorporate this in the future development.