Best approach for 2D gesture recognition in VR?

Question

I'm the developer on a game which uses gesture recognition with the HTC Vive roomscale VR headset, and I'm trying to improve the accuracy of our gesture recognition.

(The game, for context: http://store.steampowered.com//app/488760 . It's a game where you cast spells by drawing symbols in the air.)

Currently I'm using the 1 dollar algorithm for 2D gesture recognition, and using an orthographic camera tied to the player's horizontal rotation to flatten the gesture the player draws in space.

However, I'm sure there must be better approaches to the problem!

I have to represent the gestures in 2D in instructions, so ideally I'd like to:

Find the optimal vector on which to flatten the gesture. Flatten it into 2D space. Use the best gesture recognition algorithm to recognise what gesture it is.

It would be really good to get close to 100% accuracy under all circumstances. Currently, for example, the game tends to get confused when players try to draw a circle in the heat of battle, and it assumes they're drawing a Z shape instead.

All suggestions welcomed. Thanks in advance.

Answer 1

Believe me or not, but I found this post two months ago and decided to test my VR/AI skills by preparing a Unity package intended for recognising magic gestures in VR. Now, I'm back with a complete VR demo: https://ravingbots.itch.io/vr-magic-gestures-ai

The recognition system tracks a gesture vector and then projects it onto a 2D grid. You can also setup a 3D grid very easily if you want the system to work with 3D shapes, but don't forget to provide a proper training set capturing a large number of shape variations.

Of course, the package is universal, and you can use it for a non-magical application as well. The code is well documented. Online documentation rendered to a PDF has 1000+ pages: https://files.ravingbots.com/docs/vr-magic-gestures-ai/

The package was tested with HTC Vive. Support for Gear VR and other VR devices is progressively added.

Answer 2

Seems to me this plugin called Gesture Recognizer 3.0 could give you a great insight on what step you should take

Gesture Recognizer 3.0

Also, I found this javascript gesture recognition lib in github

Jester

Hope it helps.

Answer 3

Personally I recommend AirSig It covers more features like authentication using controllers. The Vive version and Oculus version are free. "It would be really good to get close to 100% accuracy under all circumstances." My experience is its built-in gestures is over 90% accuracy, signature part is over 96%. Hope it fits your requirement.