In the process of using Jellyfin, I once encountered the problem of subtitle rendering errors. Start with the source code, simply analyze the cause of the problem, and provide a simple solution.
Antecedent knowledge
Subtitles can be simply divided into hard subtitles, built-in subtitles and external subtitles.
The habitual name only represents the personal opinion within the scope of Zimiao's haunting blog (azimiao. com).
- Hard Caption
When pressing, the caption image is directly rendered on the video image.
- Internal subtitles/internal subtitles/internal subtitles
Encapsulate one or more subtitle streams in MKV and other packages, and render subtitles according to the specified timeline and style information during decoding.
- External subtitles
It is similar to the built-in subtitle, but the subtitle is a separate subtitle file that is not encapsulated in the video file.
Caption rendering of Jellyfin
Only Jellyfin Server+Jellyfin Web will be discussed. Generally, third-party clients based on WebView are similar.
Two rendering modes
In Jellyfin, subtitle rendering is divided into two modes:
- Burn subtitle image
- Front end rendering caption
Burn subtitle image Similar to hard subtitles, in this mode, when Jellyfin calls ffmpeg transcoding video, the subtitles are directly rendered to the streaming video clip.
Front end rendering caption The subtitle is rendered by the browser. The server sends the subtitle text, and the front-end plug-in generates the subtitle image in the browser. Jellyfin uses the modified Javascript SubtitlesOctopus.
Timing
When you open the video, Jellyfin will call ffmpeg to continuously generate TS stream slices; When you exit the playback interface, these clips will be deleted automatically.
For text type subtitles (S_TEXT/ASS, etc.), Jellyfin gives priority to front-end rendering, which has one advantage: for videos with multiple subtitle tracks, there is no need to regenerate TS slices when switching subtitles, and the generated TS slices without subtitles are directly reused.
When subtitle encoding is not supported by front-end rendering plug-ins (such as HDMV/PGS type subtitles), Jellyfin will use the mode of burning subtitle images.
Error rendering subtitles in burning subtitle image mode
There are few occasions to burn subtitle images in Jellyfin. The most common is to encounter HDMV/PGS and other image types of subtitles.
An adverse example, 610M weather son full special effects subtitles:
However, HDMV/PGS is originally an image based caption, which is equivalent to saving a bunch of pictures. During ffmpeg processing, the overlay video filter directly pastes the image without re reading the font to generate a font map, so there is little garbled code.
Find the TS after transcoding, and you can see that subtitle images have been mixed in the video image:
For video files with embedded fonts, when Jellyfin calls ffmpeg rendering, the fonts will be extracted to a separate folder and designated as the fontdir of the libass caption filter:
Note that the configuration is for libass: FFmpeg does not come with any fonts FFmpeg uses libass to burn subtitles, which in turn uses fontconfig to scan fonts available on system and use them.
//Jellyfin EncodingHelper.cs var fontPath = Path.Combine(_appPaths. CachePath, "attachments", state.MediaSource.Id); var fontParam = string.Format( CultureInfo.InvariantCulture, ":fontsdir='{0}'", _mediaEncoder. EscapeSubtitleFilterPath(fontPath));
For video files without embedded fonts, when ffmpeg rendering is called, libass will automatically use the System Font folder (for example, Windows/Fonts under Windows).
Therefore, rendering errors will not occur in this case.
Error rendering subtitles in front-end rendering mode
Jellyfin will send text based subtitles such as TEXT/ASS to the browser, and use the modified version of Jellyfin JavascriptSubtitlesOctopus to render. The purpose of front-end rendering is to save server resources.
For videos with internal text subtitles, Jellyfin will extract the subtitle track into a separate file and send it to the browser.
The brief workflow of JavascriptSubtitlesOctopus in Jellyfin is as follows:
- Get font list
- Get subtitle file
- Parsing subtitle files (timeline, FontStyle, FontText, etc.)
- Rendering subtitles based on video timeline and FontStyle
You can confirm the resource acquisition logic through the F12 console of the browser.
In this case, many people will find rendering errors in Chinese, Japanese, etc., and the copy will be displayed as a pile of blocks "□□□□□". This kind of garbled code problem occurs in the process of rendering subtitles, because no font file containing glyphs is found.
As a front-end plug-in, JavascriptSubtitlesOctopus cannot read the client's local font file, but asks the Jellyfin server for a list of font files. Therefore, there are two different situations:
1. Video with embedded font
When some suppression groups suppress the video, they will also encapsulate the font file into the video file, as shown in the following figure:
For videos with embedded fonts, Jellyfin will extract the fonts from the videos and send them to JavascriptSubtitlesOctopus as the fonts to be used.
# Get /Videos/[VideoUID]/[VideoUID]/Attachments/*
Encapsulated fonts generally contain all characters, so there are few garbled characters. At most, the font order is confused.
2. Video without embedded font
For videos without embedded fonts, Jellyfin returns the font list under the Fallback font folder. The rules for generating the font list are as follows:
- Sorted according to the priority of small file size/top name/latest modification time/latest creation time;
- Take the first few in the sorting list. When the total font size in the list is greater than 20M, no new font will be added.
This source code is as follows:
//Zimiao.com fork from github/jellyfin/jellyfin public IEnumerable<FontFile> GetFallbackFontList() { var encodingOptions = _serverConfigurationManager.GetEncodingOptions(); var fallbackFontPath = encodingOptions.FallbackFontPath; if (! string.IsNullOrEmpty(fallbackFontPath)) { var files = _fileSystem.GetFiles(fallbackFontPath, new[] { ".woff", ".woff2", ".ttf", ".otf" }, false, false); var fontFiles = files .Select(i => new FontFile { Name = i.Name, Size = i.Length, DateCreated = _fileSystem.GetCreationTimeUtc(i), DateModified = _fileSystem.GetLastWriteTimeUtc(i) }) .OrderBy(i => i.Size) .ThenBy(i => i.Name) .ThenByDescending(i => i.DateModified) .ThenByDescending(i => i.DateCreated); // max total size 20M const int MaxSize = 20971520; var sizeCounter = 0L; foreach (var fontFile in fontFiles) { sizeCounter += fontFile.Size; if (sizeCounter >= MaxSize) { _logger. LogWarning("Some fonts will not be sent due to size limitations"); yield break; } yield return fontFile; } } else { _logger. LogWarning("The path of fallback font folder has not been set"); encodingOptions.EnableFallbackFont = false; } }
After obtaining the font list, the customized JavascriptSubtitlesOctopus does not match the FontName, but adopts a simple and crude approach:
- If the current character can be rendered by the first font, use it to render the current character, or use the next font in the list to render;
- If the font in the whole list does not contain this character, try to use the default font to render the character (note that the resource organization form of the front-end plug-in in this version is different from that of the general version, which will be discussed later);
- If the default font still cannot render the character, an error is reported.
The following figure is an example. Although the same font name is used to mark Chinese and English in ASS, English fonts are ranked first, so English fonts are directly used for rendering, while Chinese fonts are rendered with fonts containing Chinese characters:
Clever you may have thought that since fallback is on the default font, is it better to replace the default font?
Unlike the general version of the JavascriptSubtitlesOctopus plug-in, Jellyfin's customized version of JavascriptSubtitlesOctopus packages default resources in wasm:
#Zimiao.com fork from github/jellyfin/JavascriptSubtitlesOctopus dist: src/subtitles-octopus-worker.bc dist/js/subtitles-octopus-worker.js dist/js/subtitles-octopus-worker-legacy.js dist/js/subtitles-octopus.js dist/js/COPYRIGHT dist/js/default.woff2 dist/js/subtitles-octopus-worker.js: src/subtitles-octopus-worker.bc src/pre-worker.js src/SubOctpInterface.js src/post-worker.js build/lib/brotli/js/decode.js mkdir -p dist/js emcc src/subtitles-octopus-worker.bc $(OCTP_DEPS) \ --pre-js src/pre-worker.js \ --pre-js build/lib/brotli/js/decode.js \ --post-js src/SubOctpInterface.js \ --post-js src/post-worker.js \ -s WASM=1 \ $(EMCC_COMMON_ARGS)
Therefore, things are not as simple as direct replacement. Here are three easy to think of ways:
- Replace and repackage Jellyfin customized JavascriptSubtitlesOctopus;
- The Fallback directory directly points to System Font;
- Provide a all-in-one font to put in the Fallback directory;
Practice 1 is to treat both symptoms and root causes, but every time Jellyfin upgrades, your changes will be overwritten. It is your own job.
Method 2 is not feasible. On Windows, if you follow the sorting rules above, the font list is all English characters (pay attention to the source code sorting priority).
Practice 3 is relatively easy to implement and will not be overwritten by future updates of Jellyfin.
Solution: Use the all-in-one font as the rendering font
I recommend using the open-source Siyuan boldface as the all-purpose Fallback font. Siyuan boldface contains most commonly used foreign characters in Chinese, Korean, Japanese, English, and so on, and provides Variable font (variable font, one font file can render characters of different thicknesses, italics, and other styles).
github: adobe-fonts/source-han-sans Siyuan boldface is a set of OpenType/CFF pan Chinese Japanese Korean fonts. This open source project not only provides available OpenType fonts, All source files for creating these OpenType fonts using AFDKO tools are also provided.
What is recommended by Zimiao's haunting blog SourceHanSansTC-VF , which contains characters and properties as described above.
Don't forget to specify the folder here:
If no folder is specified, Jellyfin will not return the Fallback font list (see the source code above).
Other issues
Q: I also found 610M weather son full special effect subtitles, why does Jellyfin not display?
A: At present, the stable version of Jellyfin does not recognize external sup files, and the beta version already supports them, waiting for the stable version to be updated (far away).
Q: Do you recommend updating Jellyfin to the beta version?
A: Not recommended. There are many bugs, and the old version cannot be returned perfectly.
