FFmpeg Player Theory and Practice Course

1-1

5G+audio and video "5G": The development law of the Internet of Everything audio and video industry has always been driven by demand and technological innovation, and they are mutually causal. In the AI era, technological innovation will continue to tap the potential of the audio and video industry, so that it has a stronger information carrying capacity and more potential application value, thus constantly promoting industrial upgrading. With the commercial start of 5G, what business opportunities will the audio and video industry meet? In the fields that 5G will affect, the audio and video industry is an important part. 5G: "Zero delay" and "5G+audio and video industry" will further tap the potential of the audio and video industry, accelerate the deep integration with cloud computing, artificial intelligence and other fields, and constantly promote new formats and new business models.  Live performance&church application  Museum  Public security  Command and control center  Business management In recent years, one of the most important developments in the field of IP video network is the transmission of high-quality video. Regardless of the number of domain names in the video network, it can be up to 4K/60/4:4:4+HD - close to zero delay (less than 1 millisecond).

"Only for paying users" Click to download "Introduction to Audio Video Xiaobai -- What is 1wm Audio Video? Pdf"

1-2

Puzzlement of audio and video self-study The development of audio and video technology is also getting more and more attention. However, many developers are intimidated by the wide range of knowledge involved in audio and video development and the relatively high threshold for entry. Although there are many blogs on the Internet summarizing the learning path of audio and video technology, the relevant knowledge is relatively independent, including "audio and video decoding related", "OpenGL related", "FFmpeg related", and "RTP/RTCP," RTMP, HLS, QUIC and other communication related ", but for novices, it is very difficult to connect all the knowledge and understand all the knowledge well. In the process of learning audio and video development, I deeply felt all kinds of confusion and pain caused by the dispersion of knowledge and transition fault, so I hope that through my own understanding, I can summarize the knowledge related to audio and video development, form a series of articles, step by step, and analyze each link, Second, I hope it can help developers who want to start audio and video development.

"Only for paying users" Click to download "Course learning materials: courseware source code. rar"

1-3

What is a video What is video? Video refers to various technologies that capture, record, process, store, transmit and reproduce a series of static images in the form of electrical signals. When the continuous image changes more than 24 frames (fps) per second, according to the principle of visual persistence, the human eye cannot distinguish a single static image; It looks like a smooth and continuous visual effect. Such continuous pictures are called videos. Video technology was first developed for television system, but now it has developed into various formats to facilitate consumers to record video. The development of network technology also promotes that video clips exist on the Internet in the form of streaming media and can be received and played by computers. Video frame is a basic concept of video, representing a picture, such as a page in the Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip Flip. A video is composed of many frames. Frame rate Frame rate, that is, the number of frames per unit time, in frames per second or fps. For example, in a FlipBook, how many pictures are included in one second? The more pictures, the smoother the picture and the more natural the transition.

1-4

Color space RGB and YUV Here we only talk about two commonly used color spaces.  1) RGB: RGB color mode should be the one we are most familiar with, and it is widely used in current electronic devices. All colors can be mixed through the three basic colors of R G B;  2) YUV: Here we focus on YUV, which is not familiar to us. This is a color format that separates brightness from chroma. Early television was black and white, that is, only the brightness value, that is, Y. With color TV, two shades of UV have been added to form the current YUV, also known as YCbCr. 1) Y: Brightness is the gray value. In addition to indicating the brightness signal, it also contains more green channels; 2) U: Difference between blue channel and brightness; 3) V: Difference between red channel and brightness.

1-5

What is audio The most common way of carrying audio data is pulse code modulation, or PCM. In nature, sound is continuous and an analog signal. How can we preserve sound? That is to digitize sound, that is, convert it into digital signal. We know that sound is a wave, which has its own amplitude and frequency. To preserve sound, we must preserve the amplitude of sound at various time points. Digital signals cannot continuously store the amplitude at all time points. In fact, it is not necessary to store continuous signals to restore the sound acceptable to the human ear. According to Nyquist sampling theorem: in order to restore the analog signal without distortion, the sampling frequency should not be less than twice the zuigao frequency in the analog signal spectrum. According to the above analysis, the acquisition steps of PCM can be divided into the following steps: analog signal ->sampling ->quantization ->coding ->digital signal audio? Audio is a professional term. All the sounds that people can hear are called audio. The sampling rate and sampling bit sampling rate are the sampling frequency. As mentioned above, the sampling rate should be more than twice the original acoustic frequency, and the Zuigao frequency that can be heard by the human ear is 20kHz. Therefore, in order to meet the auditory requirements of the human ear, the sampling rate should be at least 40kHz, usually 44.1kHz, and the higher one is usually 48kHz. Note: The range of human auditory frequency is [20Hz, 20KHz], which involves the amplitude quantization mentioned above. The waveform amplitude is also a continuous sample value in the analog signal, while in the digital signal, the signal is generally discontinuous, so the analog signal

1-6

Channel code rate sampling rate Number of sound channels Number of sound channels refers to the number of sounds that support different sounds (note different sounds). Bit rate Bit rate refers to the amount of information that can pass through a data stream every second, in bps (bit per second). Bit rate=sampling rate * sampling bits * number of sound channels Audio format refers to playing or processing audio files in the computer, that is, digital to analog conversion of sound files. This process also consists of sampling and quantization, The low frequency of zui ranges from 20Hz to 20KHZ, and the human ear cannot hear it above 20KHz. Therefore, the large bandwidth of the audio file format of zui is 20KHZ, so the sampling rate needs to be between 40 and 50KHZ, and more quantization bits are required for each sample. The standard of audio digitization is the signal to noise ratio of 16 bits to 96 dB for each sample. PCM is modulated by linear pulse coding, and each quantization step has the same length. This standard is adopted in the production of audio files. Common audio formats are: CD format, WAVE (* WAV）、 AIFF、MP3、MIDI、  AAC、WMA、OggVorbis。

"Only for paying users" Click to download "Introduction to Audio Video Xiaobai -- What is 1wm Audio Video? Pdf"

1-7

Fundamentals of Video Coding The coding here is not the same concept as the coding (digital coding) mentioned in the audio above, but refers to compression coding. We know that in the computer world, everything is composed of 0 and 1, and audio and video data are no exception. Due to the huge amount of audio and video data, if the raw stream data is stored, it will consume a lot of storage space and is not conducive to transmission. Audio and video actually contain a lot of duplicate data of 0 and 1, so you can compress these data of 0 and 1 through certain algorithms. Especially in the video, because the picture is gradually transitioning, the entire video contains a large number of pictures/pixel repetition, which just provides a very large compression space. Therefore, encoding can greatly reduce the size of audio and video data, making it easier to store and transmit audio and video. So, how large is the data volume of raw audio and video without coding? Take a video with a resolution of 1920 × 1080 and a frame rate of 30 as an example: 1920 × 1080=2073600 (Pixels), each pixel is 24bit; That is, each picture 2073600 × 24=49766400 bit, 8 bit=1 byte; So: 49766400bit=6220800byte ≈ 6.22MB. This is the original size (6.22MB) of a 1920 × 1080 picture, multiplied by the frame rate of 30. That is, the size of video per second

1-8

Fundamentals of Audio Coding The original PCM audio data is also a very large amount of data, so it also needs to be compressed and encoded. Like video coding, audio has many coding formats, such as WAV, MP3, WMA, APE, FLAC, etc. Music lovers should be very familiar with these formats, especially the last two lossless compression formats. However, today's protagonists are not them, but another compression format called AAC. This section takes the AAC format as an example to intuitively understand the audio compression format. AAC is a new generation of lossy audio compression technology, an audio compression algorithm with high compression ratio. Most of the audio data in MP4 video are compressed in AAC format. AAC format is mainly divided into two types: ADIF and ADTS.  1) ADIF: Audio Data Interchange Format. Audio data exchange format. The feature of this format is that it is possible to find the beginning of the audio data without decoding in the middle of the audio data stream, that is, its decoding must be performed at the beginning of a well-defined. This format is often used in disk files. ADIF has only one unified header, so all data must be obtained before decoding  2) ADTS: Audio Data Transport Stream. Audio data transmission stream. The feature of this format is that it is a bit stream with synchronized words, and decoding can start anywhere in this stream. Its characteristics are similar to mp3 data stream format.

1-9

Audio and video packaging container Careful readers may have found that none of the audio and video coding formats we introduced above are the ones we usually use, such as mp4 Rmvb, avi, mkv, mov... Yes, these familiar video formats are actually containers wrapped with audio and video coding data, which are used to mix video streams encoded with specific coding standards with audio streams to form a file. For example, mp4 supports video coding such as H264 and H265, and audio coding such as AAC and MP3. Mp4 is a popular video format at present. At the mobile end, videos are generally encapsulated in mp4 format.

1-10

What is multimedia Multimedia technology refers to the technology of comprehensive processing and management of text, data, graphics, images, animation, sound and other media information through computers, so that users can interact with computers in real time through a variety of senses. It is also called computer multimedia technology. [3.2] Real multimedia technology Real multimedia technology involves the product of computer technology, while other simple things, such as movies, television, audio, etc., do not belong to the category of multimedia technology. The media in multimedia technology mainly refers to the former, that is, the use of computers to digitize the text, graphics, images, animation, sound, video and other media information, and integrate them on a certain interactive interface, so that computers have the ability to display different media forms interactively. It has greatly changed people's traditional way of obtaining information, and conforms to people's reading style in the information age. [3.3] What is media? Media refers to the carrier of information, and its essence is the technology and means of information dissemination.

1-11

Multimedia concept, characteristics and advantages What is multimedia? The so-called multimedia refers to a computer system that can handle more than two kinds of sensory media at the same time. Its goal is to provide users with a richer application experience. Sensory media: Sensory media refers to the information form or media form that directly acts on human sensory organs and is accepted by human perception system. The sensory media mainly includes words, sounds, graphics, images, animations, videos and other forms.  The first feature of multimedia technology is the diversity of media types or media technologies. An application system that can be called multimedia must integrate at least two different types of media and related technologies.  The second feature of multimedia technology is the synchronization of media content. In multimedia application systems, multiple media are integrated together, and they work in a collaborative way.  The third feature of multimedia technology is interactive.

1-12

Fundamentals of Digital TV Digital TV is a new generation of TV which adopts digital coding and digital transmission technology from TV program recording, broadcasting to transmission and reception. It has many advantages, such as: it can realize two-way interactive services, strong anti-interference ability, high utilization rate of frequency resources, and it can provide high-quality television images and more video services (such as interactive television, distance education, conference television, television commerce, film and television on demand, etc.). Digital Television (DTV), referred to as DTV, is a television system that processes signals in a digital way from program acquisition, program production, program transmission to the user end, that is, digital signals are used in all links from the studio to transmission, transmission and reception, or transmitted through the digital sequence composed of 0 and 1 digital strings. Digital TV is the third generation TV type after black and white analog TV and color analog TV. It is a concept relative to analog TV. Compared with analog TV, digital TV has higher picture quality, stronger functions, better sound effect, richer content, and usually has interactivity and communication functions. Television digitization is another major technological revolution in the history of television development.

1-13

Basic Short Video Short video refers to the video content played on various new media platforms, suitable for watching in the mobile state and short-term leisure state, and pushed frequently, ranging from a few seconds to a few minutes. The content integrates skills sharing, humor, fashion trends, social hot spots, street interviews, public education, advertising creativity, business customization and other topics. Due to its short content, it can be made into a single film or a series of columns. [5.2] Short video is different from micro film and live broadcast. Short video production does not have the same specific expression form and team configuration requirements as micro film. It has simple production process, low production threshold, strong participation and other characteristics, and is more valuable than live broadcast, The ultra short production cycle and interesting content have certain challenges to the copywriting and planning skills of the short video production team. Excellent short video production teams usually rely on mature We Media or IP. In addition to high frequency and stable content output, they also have strong fan channels; The emergence of short videos has enriched the form of native advertising in new media.