Recently, I am interested in SteamVR media player and simply record its technical details and implementation method (by the way, see how it is different from the player I have worked on before).
previously on
This is the last article on media player analysis. The first two articles are as follows:
sketch
Generally, the process of playing video in VR player is as follows:
- Real time decoding to get frame texture
- The texture is processed according to the stereo format of the video
- Render on screen
SteamVR player is no exception.
decode
Similar to the above (2), the player decoding also highlights the simplicity of the main idea. The decoding directly uses the built-in VideoPlayer of Unity, and the subsequent operations use the texture decoded by the VideoPlayer.
Due to the use of the built-in VideoPlayer, due to its limited decoding ability, the performance of playing large bit rate video is poor.
VideoPlayer has restrictions on file suffixes. Based on this, the player has restrictions on video file suffixes and only plays .mp4 、 .avi 、 .mov Video file with suffix, Therefore, VideoPlayer cannot play even the. mkv file encoded as H.264 internally 。
At present, there are more and more video files using the ten thousand gold oil package format of MKV, and its internal video coding format is also varied. It is not a good idea to judge by suffix.
Match Stereo Format
Currently, the layout supported by SteamVR player is:
- Mono
- StereoLR/StereoRL
- StereoTB/StereoBT
- Anaglyph (red blue/Gray)
For each Layout, the following rendering methods are supported:
Note: At present, there is a special 360 ° stereoscopic video implementation mode in the industry. It uses four or eight binocular cameras to record four or eight directions of stereoscopic images at the same time. It is spliced according to the user's viewing angle, which is expensive and requires special software to play. It is not the above general stereoscopic video format.
In addition to format control through the UI in (I), users will perform automatic stereo format matching for videos that have not been played.
The logic of automatic stereo format matching is to first use the path and file name for keyword matching, if not, then match according to the file width height ratio.
Because there are many path matching keywords, the list here is too long, so only the first five keywords are listed:
| Text |
Layout |
Format |
| _2dp |
Mono |
2D |
| _3dph |
Mono |
LR |
| _3dpv |
Mono |
TB |
| _180x180_3dh,_stereo180 |
one hundred and eighty |
LR |
| There are many more, just |
|
|
After the path keyword matching fails, the corresponding relationship of the width to height ratio matching format is:
| Width |
Layout |
Format |
| <1.5 |
one hundred and eighty |
LR |
| >=1.5 |
three hundred and sixty |
TB |
According to the aspect ratio, most of the matching is wrong, and the user needs to manually change the stereo format through the UI.
After the change is completed, the information will be stored for reading in the next play.
play
For each logic frame, through Graphic Blit copies the frame decoded by VideoPlayer to a texture created by itself.
There are two groups of planes in the scene, one group of planes and one group of spheres. The plane is used for Mono Layout playback, and the sphere is used for 180, 360, and fisheye Layout playback.
There are two faces in each group of planes, corresponding to the left and right eyes respectively.
At the beginning of playback, send the texture created by yourself and the supporting parameters to the four face Shader for sampling.
Because the Shader is DXBC, common tools cannot solve it directly, but can only deduce it based on bytecode, so no more research is needed.
However, the logic of Shader should not be complicated, because it is nothing more than sampling according to the video stereo format, which is not so complicated.
end
This completes the SteamVR player analysis article.
To sum up, the SteamVR player project is simple and complex. The simplicity is reflected in its implementation ideas are not beyond the engine's own capabilities, while the complexity is reflected in its control over Native (see article 2).
After the analysis is completed, it is planned to reprint a player: the reprint player also supports mouse+VR handle operation, also has Desktop UI and VR image UI, and uses modern player plug-ins to replace VideoPlayer.
