Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - VirtualGrasp (Speeding up the virtual reality revolution with realistic & real-time animation of hand-to-object interaction)

Teaser

• What is the problem/issue being addressed?Challenge—animating hands in VR in a realistic way is an extremely complex task.Hand presence isn’t just about accuracy of the position of the hand in space, but also about representing the myriad subtleties that our endlessly...

Summary

• What is the problem/issue being addressed?
Challenge—animating hands in VR in a realistic way is an extremely complex task.
Hand presence isn’t just about accuracy of the position of the hand in space, but also about representing the myriad subtleties that our endlessly adaptable hand joints & finger digits are capable of.
Solution— The VirtualGrasp by Swedish SME Gleechi is the 1st & only commercially available software that creates realistic generic animation of hand movements & their interactions with any kind of objects while being fully hardware agnostic. VirtualGrasp replaces labor-intensive manual animations of the hands by using predictive & adaptive machine learning algorithms that analyze the physical properties of a virtual object, model the most appropriate & realistic grip formation for the hand & snap the hand to that position.

• Why is it important for society?
VirtualGrasp is a core technology that will speed up the VR revolution. As VirtualGrasp provides realistic animations of hands & hand-to-object interactions, it will enable VR applications that were not possible (or, in other words, not convincing enough to be useful) before.
In reality we are most intimately familiar with the movements of our hands (since they are in our line of vision most of the time)—we will instantaneously detect any discrepancies in hand movements that will render the VR simulation not immersive enough.

• What are the overall objectives?
A - Training in manufacturing
Problem—The average training time in manufacturing before new employees are fully productive is 1-2 years. While learning by doing has been proven as the most efficient way to learn practical tasks, staff cannot practice on many machines due to the high costs & dangers.
Solution—VirtualGrasp allows machine operators to practice hands-on training in a safe virtual environment and to train emergency responses. It also allows to stage in real-life expensive and difficult-to-simulate situations . By introducing a standardized technology to enable interactions for training purposes, Gleechi can significantly reduce the time it takes to create VR training applications for customers. Further, VirtualGrasp has been proven to increase the memorization ability from learning tasks in VR.

B – Stroke rehabilitation
Problem—One of the key reasons for why people never recover from stroke is because of lack of motivation to travel to rehabilitation centers and perform repetitive rehabilitation exercises. Stroke rehabilitation is moving from hospital-based therapy sessions to home-based remote sessions (telerehabilitation) using 3D cameras and gloves.
Solution—Although competitive approaches in this field exist, VirtualGrasp is the only solution available for visualizing for fine finger movement and accurate grasping. With VirtualGrasp, limited movement from patients’ hands will be amplified to make them able to use their virtual hands perfectly. Clinical studies have shown that visually amplified movement can significantly improve rehabilitation45. By enabling stroke rehabilitation for the hands, we will enable millions of people who suffers from stroke every year to perform more convenient stroke rehabilitation from their home & return to normal life & work.

C – 3D videogames
Problem—Hands are notoriously difficult and time-consuming to animate in 3D videogames, and many of the developers settle with unrealistically looking hands, with fingers protruding objects or hide hands altogether.
Solution—VirtualGrasp solves the problem of accurate hand visualization and hand-to-object interaction and makes it affordable not only for large game developers but to a long tail of smaller independent game vendors.

Work performed

\"3D Videogames - We have executed the trial case #1 - testing VirtualGrasp with a VR videogame developers: Red PIll (USA). We have both been integrating VirtualGrasp plugin into the existing game engines and providing the SDK for developers to implement VirtualGrasp themselves. We have also improved VirtualGrasp to adapt it to new customer needs identified during the trial.
The trial case was focused on a DJ application, a VR-based interactive virtual DJ application to accurately replicate the motion of hands in fingers interacting with turntables and mixer that the company is developing. Red Pill has identified that high-quality animations are a must-have for the application to be successful. See deliverable 3.1, 3.2 and 3.3.

Training in manufacturing - We have executed the trial case #2 —application of VirtualGrasp to manufacturing training in VR. This work package includes implementation of VirtualGrasp in the training application, evaluation of the results, improvement of the technology, and preparation for commercial exploitation in simulation environments for manufacturing. See deliverable 4.1, 4.2 and 4.3.

Stroke rehabilitation - We executed the trial case #3 – applying VirtualGrasp to validate and test the progress of stroke rehabilitation for hands & fingers with the help of VR.
Gleechi aims to initially provide a fully packaged rehabilitation application built in Unity Engine for Windows, and the interaction in the application is generated by the VirtualGrasp software. See deliverable 5.1, 5.2 and 5.3.\"

Final results

VirtualGrasp is based on 8+ years of research in robotics by two of Gleechi’s co-founders, Dan Song (PhD in biomedical engineering) & Kai Hübner (PhD in computer science & robotics). While working with robots, Kai & Dan were working on a notoriously difficult problem—robot hands and grippers are not capable of picking random objects, limiting robots to performing simple and pre-programmed tasks. This is caused by the current state of computer perception which does not yet allow building an accurate model of any random object so that the robot could grasp it in a correct way without damaging the object.
The same problem of animating hands turned out to exist in VR development, but with a much higher degree of urgency & much higher potential for commercialization. In VR, the problem of hands correctly interacting with random objects is the same as in robotics, however there is much more information about objects (such as 3D object meshes) readily available. They created VirtualGrasp, which enables realistic hand animation & hand-to-object interaction, and has applications in diverse fields—teaching surgery, simulating working environments in manufacturing, stroke rehabilitation for paralyzed hands,
and, of course, 3D videogames—which have been confirmed by the Gleechi’s prospects.

The key 2 benefits of VirtualGrasp for VR software developers and their users are:
For VR software developers— Real-time realistic hand movement animation & interaction with objects: Enabling more complex VR environments & richer object interaction scenarios (especially in healthcare, manufacturing & 3D videogames). Enabling truly immersive user experiences & complete freedom of exploration of the VR object space.
For end users—Reduced cognitive burden - Allowing users to intuitively interact with objects, using their natural hand movements to grasp & hold the objects in VR. By visualizing realistic grasps, VirtualGrasp removes the cognitive load that comes from seeing own hands behaving in non-natural ways, which reduces the mental effort and does not demand visual focus. This effect has been proven to enhance the memorization ability for users that perform training tasks in VR significantly.

Website & more info

More info: http://www.gleechi.com.