Recently, I am interested in SteamVR media player and simply record its technical details and implementation method (by the way, see how it is different from the player I have worked on before).
This is the first note, which mainly describes the implementation of UI rendering and interactive event response of SteamVR media player.
This article only discusses the rendering of UI. As for video 3D rendering, video 2D window image, file list, etc., I will write it later.
UI display and rendering
This is the Desktop 2D window view (left) and 3D view (right) in VR of SteamVR media player.
Although it seems that there are two sets of UIs with different interaction modes and display dimensions (namely, one set of 2D window UI and the other set of VR 3D space UI), there is only one set of 2D UI in the actual program.
Of course, this 2D UI is made with UGUI.
How to display a UI in 2D window and 3D space at the same time?
Decompiling shows that the media player will copy the drawn content to the RT of 3D space patch through simple Blit at the end of the execution of Unity Player's logical frame+rendered frame:
//Called after WaitForEndOfFrame //=Execute these codes to copy rendering results when logical frame+rendered frame is completed this.m_lastFrameGuiTextureCommandBuffer = new CommandBuffer(); this.m_lastFrameGuiTextureCommandBuffer.name = "m_lastFrameGuiTextureCommandBuffer"; this.m_lastFrameGuiTextureCommandBuffer.Blit(BuiltinRenderTextureType. CurrentActive, this.m_lastFrameGuiTexture); Graphics.ExecuteCommandBuffer(this.m_lastFrameGuiTextureCommandBuffer);
The Blit of CommandBuffer and that of Graphic can be understood as the same function, both of which are used to copy content.
Using OpenGL analogy, Unity Blit is equivalent to the following three steps:
glBindFramebuffer(GL_READ_FRAMEBUFFER, CurrentActiveFBO); glBindFramebuffer(GL_DRAW_FRAMEBUFFER, lastFrameGuiTextureFBO); glBlitFramebuffer(0, 0, srcWidth, srcHeight, 0, 0, dstWidth, dstHeight, GL_COLOR_BUFFER_BIT, GL_NEAREST); // glBindFramebuffer(GL_READ_FRAMEBUFFER, 0); // glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
Then someone will ask why the copied current Target does not have the images of the left and right cameras?
This question is easy to answer:
Most VR helmets are directly displayed on the XR SwapChain image when the left and right eye cameras render.
Although some VR SDKs make 2D windows display some content by default (for example, the old Steam VR SDKs default to LeftEye or RightEye images, some third-party SDKs default to black screen configuration, and the XRSettings.gameViewRenderMode API of the new Unity), these contents, even if any, are covered by a full screen and opaque UI.
Therefore, a simple CommandBuffer copies the 2D UI rendering results to the 3D patch without any loss (only one more RT, one GPU copy, negligible).
But this also has a small disadvantage: 3D UI patch refresh in VR will be 1 frame slow. The reason is simple: we update the 3D UI patch RT after the end of the rendered frame.
However, since the rendering frame rate of VR is 72 or more frames, small shortcomings are harmless.
User interaction of UI
Compared with other VR players, the biggest interactive advantage of SteamVR media player is that it can be operated by VR handle or mouse, and the mouse arrow position will be displayed in VR during mouse operation.
Mouse operation and event distribution (2D window UI interaction)
Go directly to the set of EventSystem+InputModule+GraphicRaymaster event distribution by default of Unity, without further explanation.
VR handle operation (pseudo VR 3D UI interaction)
The common 3D UI adaptation method of the third-party VR SDK is to write a set of CustomXRInputModule+CustomXRGraphicRaycaster in 3D space.
However, the 3D UI of SteamVR media player is not a real UI Canvas, but a meaningless mapping patch, so CustomXRInputModule+CustomXRGraphicRaycaster is not helpful here.
Valve's solution is very simple. Make a Raycast of Pose and patch, convert the intersection coordinates to Screen coordinates, call the interface in Windows user32.dll, force the Windows mouse position to be set, and distribute mouse events according to the handle key information:
[DllImport("user32.dll")] [return: MarshalAs(UnmanagedType.Bool)] private static extern bool SetCursorPos(int X, int Y); [DllImport("user32.dll")] private static extern void mouse_event(int dwFlags, int dx, int dy, int dwData, int dwExtraInfo);
This operation is equivalent to simulating mouse movement and click behavior, and UI event triggering naturally follows the above mouse event distribution process.
It has to be said that although this method is simple and crude, it perfectly solves the problem.
The mouse position is displayed in VR
The position of the mouse will be displayed on the VR patch, which is also very simple.
There is an interface in user32.dll of Windows to get the mouse position:
[DllImport("user32.dll")] [return: MarshalAs(UnmanagedType.Bool)] private static extern bool GetCursorPos(out MouseOperations. MousePoint lpMousePoint);
Convert the mouse position twice. First, convert the mouse from the Windows screen coordinates to the coordinates under the UICanvas coordinate system, and then from the UICanvas coordinate system to the coordinates under the 3D patch coordinate system.
The coordinates are already available. Then create a mouse texture patch, limit the coordinate boundary conditions, and update its position every frame.
A derivative phenomenon:
When the mouse is moved out of the media player window or the media player window is covered by other applications, the mouse can still be seen in VR, but at this time, mouse clicking and other behaviors do not respond.
This is because the mouse position is obtained through user32.dll, mapped and calculated, and always has a value; Windows will only send mouse events to the focused window, so the application cannot receive mouse operation events.
