preface
Many Xiaobai users have a misunderstanding: when they see some VR (or MR/XR/various R) manufacturers promoting gesture tracking, they always think that it can be used seamlessly in all VR applications in stock, just like the PC mouse pointer.
However, unlike 2D windows, there is no (nor can there be) a global "mouse pointer" between VR applications and VR OS (Volume 3D applications of Apple Vision Pro have a unified concept of interaction, but it is far from enough).
In 3D space, the interaction within each VR application is implemented by the application itself. If the application does not implement the corresponding interaction (such as gesture tracking), the user cannot use the interaction.
When it comes to VR/MR and gesture tracking, an unavoidable interaction is hand model grasping objects. This article will talk about the interactive realization of grasping objects freely in 3D space without wearing models.
Grab without touching?
The non touch grabbing discussed in this article mainly includes:
- Empty hand mobile hand model does not touch
- Fingers do not penetrate objects when grasping objects with one hand/two hands
- True · Free grip（ real time Generate grasping joint posture)
- Fake · free grip（ Presets Grab point and joint posture)
- Move without touching after grasping the object
- special
- Multiplayer game
This article only talks about the idea of implementation, not specific code. In the process of project implementation, it is often necessary to customize a set of interactive system according to the needs, so it is unlikely to have an interactive system that includes everything and is also universal.
Empty hand movement without threading
In VR environment, virtual world hand model is driven by Pose reported by hardware device (or gesture recognition).
There are no such obstacles in the virtual scene in the real world, so they cannot block the user's hands/handles in the real world.
In this case, if you want to move the hand model in the virtual world without wearing the model, the model Pose and the Pose reported by the device must be separated: the virtual model Pose pursues the real Pose of the device, but there can be a deviation between the two, because the model Pose can be blocked by virtual obstacles.
Two common implementations:
- Use rigid body characteristics to calculate and apply force or control rigid body speed, so that the hand model can always "chase" the real Pose;
- Using the characteristics of rigid body+joint, connect the real Pose with the hand model through the joint, and the engine will automatically complete the "chase" solution.
At a reasonable moving speed The rigid body in the physical system can be blocked by other collision bodies, so the hand model has the characteristic of non penetration.
In order to optimize the presentation effect, most games will reset the position of the hand model when the distance between the hand model Pose and the real Pose is too large (or the joint force reaches a certain threshold).
True · Free grip
principle
When grasping, true free grasping requires real-time generation of grasping joint gestures based on object Pose and hand Pose.
In VR games, true free grip is uncommon because it is difficult to control the grip effect. However, it is more realistic.
A general implementation idea:
- Obtain the most suitable HitPoint through a group of ray projection or other Cast methods;
- Move the hand model to the point closest to HitPoint but not penetrated (eg: move the hand model to HitPoint and back a distance according to the results of Physics. ComputePenetration)
- Stepping each finger until the joint touches the object or reaches the maximum bending threshold
- Logically, it is believed that the grasping has been completed at this time, and the object Pose is controlled through self calculation, joint, etc
In general, this method can obtain a more natural grasp attitude, and can grasp the object at any position.
Grasp with both hands
The grasping posture generation logic is the same as that of a single hand, which generates grasping posture through joint stepping.
However, there are differences in the influence of single/two handed grasping on object Pose: when grasping with one hand, object Pose depends on one hand Pose; When grasping with both hands, the Pose of the object depends on the Pose of both hands.
If the hand model is connected with an object through a joint, the object attitude is calculated by the joint, and the joint parameters need to be controlled;
If it is calculated by itself, it can be realized by itself according to different needs: if the design is reasonable, it should not be difficult to calculate the "position where the object should be" according to the Pose of both hands. After obtaining the position, move the object and the hand model.
Move without touching in grasping state
Similar to the empty hand movement above, the hand model and the grasped object are regarded as the same object, and can be viewed as a whole.
In some games, the object will be released when the resistance is too large and the distance between the hands is too far when the hands are grasped, so the implementation method needs to be taken as appropriate.
Fake · free grip
principle
This is a common implementation method for VR applications, and the principle is simple:
- Bind several grip points to an object, and bind each grip point to the preset grip Pose;
- When grasping, select the nearest grip point, and the hand model switches to the preset grip Pose at the grip point to complete the grasping.
Since it is the preset grip point and Pose, it will not wear the mold naturally.
When you set enough grip points, it can be approximated as free grip.
As for how grasp is executed, each has its own method: some games are directly set as sub objects, some games use joints, and some games calculate by themselves.
Grasp with both hands
There is no essential difference between two hands and one hand in selecting the grip point. The two hands can select the nearest grip point respectively.
Other precautions are the same as true free grip.
Move without touching when grasping
Same as above.
special
For virtual objects such as door handles and switches, they are no longer objects following, but hands following objects.
In this case, when writing logic, our object script needs to be extensible, at least able to perform different behaviors according to the characteristics of the object.
Multiplayer game
Personally, I think it is best to use the pseudo free grip mode in multiplayer games, because its grip results are certain and very friendly to synchronization.
MR/VR multiplayer games are either competitive or social games. The objects that can be grasped in these games are often very limited and controllable. True free grasping has no advantage (imagine that users can't hold weapons well).
