Exploration and Practice of NetEase Yunxin in the Converged Communication Scenario - SIPGateway Service Architecture - Real and Stable Personal Space of NetEase Yunxin - OSCHINA - Chinese Open Source Technology Exchange Community

Exploration and Practice of Netease Yunxin in the Converged Communication Scenario - SIPGateway Service Architecture

Author: Zhu Zhenhao, technical expert of NetEase Cloud Information, Guan Shifu

1 Background

At the beginning of 2020, the outbreak of COVID-19 epidemic and the overall economic downturn still brought huge business opportunities to some fields, especially in the field of real-time audio and video, more and more enterprises chose RTC cloud conference for communication and collaboration 。 Because of the market foundation of traditional video conferencing in the past decade, many enterprises are still using traditional hardware conferencing systems (PSTN/SIP protocol access) provided by Polycom, Cisco, etc. RTC cloud conferencing may not completely replace hardware conferencing systems in a short time, but in the current market environment, Convergence communication must be a major demand that more and more enterprises tend to choose 。

2 Comparison between hardware video conference and RTC cloud conference

Let's first take a look at how hardware video conferencing and RTC cloud conferencing are realized through which specific technologies, and what are their advantages and disadvantages.

Hardware video conference generally adopts MCU architecture (MCU: one of the multi person communication architectures, multi-point control unit, which is characterized by the server mixing all uplink media streams into one path and transferring them to the receiver), PSTN/SIP standard (SIP protocol: short for Session Initiation Protocol, protocol standard in the communication field) protocol access, which is complex in deployment and expensive in hardware cost, but can save bandwidth and terminal processing capacity.

RTC cloud conference takes NetEase conference as an example. It uses SFU architecture (SFU: one of the multi person communication architectures. Select the forwarding unit, which is characterized by The server forwards each subscribed upstream stream to the receiver, instead of mixed streams ）, private signaling access, using the NERTC (NERTC: NetEase Yunxin's self-developed audio and video communication scheme) audio and video coding system to achieve a better audio and video effect experience, and supporting Simulcast (Simulcast: multi stream transmission, that is, allowing the same terminal to send video streams of different resolution levels at the same time) flexible deployment, However, the bandwidth is large and the terminal processing capacity is required.

So how to realize the interconnection between hardware video conference and RTC cloud conference is an urgent problem to be solved. Netease Yunxin has also carried out a lot of exploration and practice in the context of integrated communication, Explore the way to realize integrated communication through SIP Gateway , and applied in Netease conferences. We will come today Share the practice of NetEase Conference on SIP Gateway server architecture in the context of integrated communication 。

3 SIPGateway

Architecture diagram

The figure shows the architecture of the SIP Gateway server, which is mainly divided into several modules:

SIP protocol access module: used to realize the access of SIP users and support RTP/RTCP protocol;
Yunxin RTC protocol access module: used to realize the access of Yunxin RTC users and support the private protocol of Yunxin RTC;
Conference management module: mainly maintain the status of rooms and participants, conference number management, etc;
Encoding and decoding module: mainly deal with audio and video encoding and decoding and transcoding;
MCU module: SIPGateway realizes the function of mixing sound and screen. Since SIP standard terminal does not support the ability of Simulcast, nor does it support receiving multiple streams, it needs to do mixed screen mixing on SIPGateway and send it to SIP terminal;
SFU module: support pure forwarding mode or select forwarding mode according to voice excitation;

How can we use the SIP Gateway server to realize cloud conference Take a look at the actual application in two specific application scenarios 。

Instance application scenario

Scenario 1: NetEase conference APP initiates the conference and supports SIP terminal to join NetEase conference.

As shown in the figure above, when the conference is initiated by NetEase Conference APP, NetEase Conference will synchronously create a SIP conference short number, and the user can join the conference after entering the short number on the SIP terminal.

The specific SIP login process will not be described too much, but can refer to the SIP standard protocol (detailed protocol link is provided at the end of this article).

Let's see that in such a scenario, the specific media stream direction is as follows:

After the SIP user logs in, the SIP Gateway synchronously creates the room and manages the user status;
When the user joins the conference, it is a SIP user, SIPGateway will simulate this SIP user as an RTC user and then join NetEase conference This is the NERTC process. After establishing the uplink RTCSession with the media server, you can push the stream, The upstream stream of the SIP end will be copied to the media server while the MCU mixed screen mixing is performed on the SIP Gateway The advantage of this is that the media service does not need special processing for SIP users, thus reducing its intrusion;
When RTC users log in to the conference, SIPGateway simulates the RTC user as a SIP user and adds it to the room on the gateway At the same time, it establishes a downlink RTCSession with the media server to receive the stream, and the media stream of RTC users is forwarded to SIP users through the mixing screen;

In this process, you need to encode and decode audio and video:

In terms of audio, PSTN/SIP generally uses G.722 or G.729 codec, while the NERTC system uses OPUS codec, so the audio needs transcoding;
In terms of video, MCU function is introduced. The process will have a certain impact on the server performance, but the overall performance meets the expectations;

The final screen layout is shown in the following figure:

Scenario 2: The conference is initiated by the traditional video conference, and Netease conference APP is supported to join the hardware video conference

As shown in the figure, the specific process: when a conference is initiated by a traditional video conference, the conference management pre system will create a hardware video conference room, RTC users input the hardware video conference number, The system will synchronously create NetEase conference number and SIP conference short number, and notify SIPGateway, which plays the role of signaling protocol and SIP intelligent routing Let RTC users communicate with hardware video conference media.

Three conferences are involved in this process: hardware video conference room, NetEase conference room and gateway conference room. The specific media stream steering and data forwarding strategies are as follows:

When the hardware video conference creates a room, NetEase conference APP will call the hardware video conference number, and create NetEase conference room and SIP gateway SIP conference short number synchronously;
After creating the SIPGateway room, the screen will be fused in the gateway conference room;
When the user joins the conference, it is an RTC user. SIPGateway will simulate a SIP user to join the gateway conference room, and establish a downlink RTCSession with the media server to receive streams;
The overall strategy of SIPGateway is Through smart signaling conversion, all users in the hardware video conference are simulated as one RTC user to join NetEase conference All RTC users are simulated as a SIP user to join the hardware video conference;
SIPGateway pushes video images, According to the energy routing strategy, select the RTC user video stream with the largest sound energy , fuse the screen and send it to the hardware video;
SIP layout screen can be set by hardware video conference control, and can be changed by switching between SIP main venue and branch venue and polling, and can also selectively view the screen of one RTC or all RTC participants;
RTC joins the hardware video conference scene to achieve real conference interconnection. SIP end users can join both hardware video conference rooms and NetEase conference rooms;

The final screen layout is shown in the following figure:

The above is an introduction to the architecture of the SIPGateway and the specific technical implementation of the two application scenarios. Let's take a look at the main features of using the SIPGateway and the supported deployment methods.

4 Features and deployment of SIPGateway

Features of SIPGateway

The use of SIPGateway has the following characteristics:

Flexible incoming call mode

Support PSTN/SIP end to actively join NetEase conference. NetEase conference also assigns a corresponding SIP conference short number to the SIP terminal when it creates the conference. SIP users can join NetEase conference only by entering the SIP conference short number;
Support inviting PSTN/SIP end to join NetEase conference. During the conference, the host selects the SIP terminal in the address book and actively invites him to join the conference;
Support NetEase conference APP to join the third-party video conference system;

Full platform interoperability

Support interoperability with iOS, Android, PC and Mac platforms;
Support PSTN/SIP terminal access;

MCU cascade

Support the cascading of SIPGateway MCU and traditional hardware video conference MCU. This scenario can support the implementation of multiple canvases.

load balancing

The load balancing module will allocate the appropriate SIPGateway according to the actual load of the SIPGateway, provided that the same room is allocated to the same SIPGateway.

High security

IP whitelist: The proxy at SIPGateway is the proxy for the SIP terminal's membership request. To prevent malicious attacks, we set an IP whitelist for the tenant to prevent illegal IP access;
Regional isolation: different tenants can route to the same SIPGateway, or to a special SIPGateway or cluster;
For enterprises with high network security requirements, SIP proxy is deployed in the DMZ area of the enterprise, which serves as the protection of the enterprise intranet, prevents intranet attacks, and can also communicate with the external network;

High availability

SIPGateway adopts cluster deployment. When one SIPGateway goes down, audio and video calls of other users will not be affected;
SIP Proxies are kept alive through Keep Alive. When one proxy goes down, it will switch to the other proxy, which will not affect the SIP call status and user audio and video calls, so as to achieve sensorless switching;

Data monitoring platform

Data platform: collect the online status, traffic status and network status of SIP users on a regular basis, which helps to locate and analyze problems in time in case of abnormalities;
Monitoring platform: automatically monitors the status of the server. When the service is abnormal, an alarm will pop up automatically, which helps to solve the service exception problem in a timely manner;

Deployment mode of SIPGateway

In terms of deployment mode, SIPGateway adopts container deployment, Multiple ways to support public cloud or privatization , flexible and changeable, to meet the needs of different scenarios.

5 Conclusion

This paper mainly shares the scenario of using SIPGateway to achieve converged communication in Netease conferences, and analyzes the specific media stream steering and data forwarding strategies through two example application scenarios.

With more and more application scenarios, such as the enterprise's internal APP mobile workbench, system integrated telephone call function, intelligent hardware, such as intelligent access control, intelligent robots, etc., higher requirements will be put forward for the interoperability of all terminals. Netease Yunxin's exploration on this track will continue to meet the needs of users in different scenarios, It really helps users grow inside.

Relevant terms mentioned in the text and derivative reading:

MCU : One of the multi person communication architectures, multi-point control unit, which is characterized by the server mixing all the upstream media streams into one path and transferring them to the receiver;
SFU : As one of the multi person communication architectures, the forwarding unit is selected, which is characterized by that the server forwards each subscribed upstream stream to the receiving end, instead of mixing streams;
SIP protocol : short for Session Initiation Protocol, the protocol standard in the communication field;
RTCSession: RTC streaming component of NetEase Yunxin;
Simulcast : Multi stream transmission, that is, the same terminal is allowed to send video streams of different resolution gears at the same time;
NERTC: audio and video communication scheme developed by Netease Yunxin;
SIPGateway: SIP protocol access gateway;

To learn more, welcome click

Ma Nong Little Fatty Brother 2024-05-16 14:40

I give you six seconds. I give you six moves with the same effect in the martial arts contest, which shows the invincibility and confidence of the master

One code Yma 2024-05-09 09:58

Recently, I often go to interviews. People who hate Ali background most regard me as a fool, even though I am a fool

CodeDoger 2024-05-02 20:48

35 It's too old to go to work and too early to retire at 60

MrChen89 2024-04-29 09:18

There are a group of people like this. I don't know what they have experienced. When it comes to HW, I can't say anything good, even if it's neutral

Francesca 2024-06-09 13:21

But the end of closed source must be open source, because many people who are dissatisfied with closed source have created open source, so the end of open source is not necessarily closed source, but to find a business model that is open source= Free Admission

zzeric 2024-04-28 20:01

Although France is the parent community, the core developers of OCCT on github are all Russians. Without Russians, the French parent community cannot continue to operate. So Huawei took over, moved to China, changed its name and resumed open source and community operations. What's the problem?

One code Yma 2024-05-06 09:14

My technical article was moved by CSDN. Why didn't anyone step on the sewing machine? This kind of report is a joke to me. The monsters with background are fine, and the monsters without background fight to death

Wang Zheng 2024-06-08 09:46

You said, "All the tests are graduate students" and smiled. I don't know my level is low.

osc_566335 2024-04-28 14:44

This is also called floor washing? Does it mean that Tesla will not wash the floor if it releases all the source code? Some people HWptds? That is to say, the language is ambiguous, which will also rise to the washing ground? Are some people too focused? Think the people he pays attention to must be staring at?

osc_27546117 2024-06-09 22:36

I have learned electric programming and expect its progress

Francesca 2024-05-19 18:00

Wine runs the Android emulator of Windows. Chrome OS is installed in the Android emulator. Linux environment is installed in chrome OS. Linux environment is installed in the Linux environment. Wine is installed in the Android emulator

kakai 2024-05-10 10:21

The world only knows that Android was created by Google. Several people know that Android is only a product acquired by Google. Similarly, what is the problem with Huawei's contribution to the collection of OGG open source work and integration into its own proprietary product line?

-SORA- 2024-04-30 17:07

When this happened in a foreign country, the comment area suddenly became very objective and rational**

pan3793 2024-06-07 22:26

Let AI give AI a score

zhangleijie 2024-06-08 10:08

pretty good

iVista 2024-06-10 18:13

I was blinded by the math test

SnailJob 2024-06-09 09:13

Yes, please continue to follow Snail Job

Happy LeapFrog 2024-05-18 09:18

But the question is: "What's the use of this for ordinary Android users?" Now the answer seems to be: "Almost nothing.".

zhuzhua 2024-05-21 10:08

I'm laughing to death. Those who have been deeply kidnapped dare not pay? Who will use the domestic open source framework of small companies in the future will be 213!!! Wait for harvesting later

brucepapa 2024-06-09 21:02

I also have several backaches... After a few days of exercise, it will be much better to focus on stretching the back muscles.

Ding Yun H 2024-06-07 20:44

There is no querydsl. Since querydsl was used, I can't look at other forms anymore

Single structure 2024-05-11 10:09

Selected as Open Source China's disgrace pillar

Bright 2024-05-19 23:25

What a fool! I killed myself. How can people deal with me later.

Xiaoxia cat ball 2024-06-09 21:29

Very good, come on

Shuimu Yi'an 2024-05-20 09:58

The news should be read continuously. I'm waiting for the third news besides rustdesk and teamviewer. Localized remote desktop software is far ahead.

Monkeys think of apes 2024-05-31 18:31

You can cheat your brother. Just don't cheat yourself

Xiao Xu Middle aged 2024-06-08 12:43

Do AI functions need networking? Will it be 404?

infoworld 2024-05-11 15:12

Universities should use open source free software instead of commercial ones. In this way, hands and feet will not be tied technically.

Xiao Xu Middle aged 2024-06-08 10:12

First place in making money!! Money and treasures will be plentiful

zoujiaqing 2024-06-07 21:22

I dare not use it

Kevin586 2024-06-08 14:41

Dream is garbage, which can also be listed and refresh my cognition

gamedot 2024-05-17 11:14

Old Zhou is deeply concerned about Huawei's great cause of open source. He is not a Huawei person, but has Huawei's soul.

zoujiaqing 2024-06-07 21:21

Spring boot was not updated last year

kangert 2024-06-09 20:07

The problem of docker hub is very uncomfortable

muwanqing123 2024-06-09 08:28

Bullshit authentication

zhy 2024-05-16 13:16

At the end of Shannon is Nong

generation

Code e person 2024-06-09 10:03

Prepare the next project and try it

Qin Liming 2024-05-11 09:12

be devoid of any sense of shame

yh2216 2024-06-09 23:03

I remember saying that one year C++was the language of the year,

Chief taxi captain 2024-05-17 11:17

I suggest that 360 open source all its products, and then become the leading enterprise in the domestic open source industry through open source, leading everyone to compete with foreign enterprises

sunday12345 2024-05-15 18:31

What does the bank do? It's blamed on the remote desktop. Persimmons really pick up soft pinches~?

kangert 2024-06-09 20:10

Really need to practice

Xiao Xu Middle aged 2024-06-10 07:05

Learn

Yeah, for 2024-05-17 13:42

That's too right. Old Zhou can't control Google, but he can control 360. Do not do to others what you do not want. All 360 products should be opened first.

Li Yinghui 2024-05-09 16:40

Buddhism has a good word, evil opinion. In dealing with the world, it is meaningless to draw conclusions from preset positions; It is also important to receive good logic training.

abeet 2024-06-08 20:38

There are no pictures, for fear that we will learn, right

lyh97157268 2024-06-09 20:58

Like c++

monkey_cici 2024-05-09 00:25

My I9 CPU, 64GB memory module and 3080Ti computer are inferior to the top configuration of 19999 on a tablet

golyu 2024-06-10 14:45

If only this was the library of solidjs

H Fine water and long flow H 2024-06-10 09:39

I haven't heard about whether fartran has paid. I'm in the top ten

yh2216 2024-06-09 13:15

Small and beautiful software development 2024-06-08 23:03

It's mainly about waist training

Xiao_f 2024-06-07 22:59

One thing to say, compared with other domestic manufacturers, Qwen's relaxed licensing fully demonstrates the style of a large factory

Ai East 2024-06-10 19:11

Absolutely easy to use

Yoona520 2024-05-17 16:34

Zhou Hongyi is now living more and more like a clown. If he stays behind the scenes, he has to become an online celebrity. Can you learn from Lei Jun?

osc_92224065 2024-04-29 10:57

Long term oppressed outsourcing of state-owned enterprises

GDWhisperer 2024-05-15 17:23

I transferred tens of thousands of yuan to my own account, which was under risk control. How did I do this? The bank should be responsible for this**

oldpig 2024-04-28 09:59

”Huawei contributed all the source code "?, the title is completely inconsistent with the content.

xiaoqibabby 2024-05-15 17:36

The bank is strongly required to be responsible for

Francesca 2024-06-10 16:19

Be ignorant. This thing has a long history. It is used for scientific computing and has high performance

Exploration and Practice of Netease Yunxin in the Converged Communication Scenario - SIPGateway Service Architecture

1 Background

2 Comparison between hardware video conference and RTC cloud conference

3 SIPGateway

Architecture diagram

Instance application scenario

4 Features and deployment of SIPGateway

Features of SIPGateway

Flexible incoming call mode

Full platform interoperability

MCU cascade

load balancing

High security

High availability

Data monitoring platform

Deployment mode of SIPGateway

5 Conclusion

Hot content

Popular comments of the whole site

About the author

Author's Album

Author's other popular articles

Hot News

Hot software

OSCHINA Community

Online tools

Introduction

QQ group

Public account

Video number

Exploration and Practice of Netease Yunxin in the Converged Communication Scenario - SIPGateway Service Architecture

1 Background

2 Comparison between hardware video conference and RTC cloud conference

3 SIPGateway

Architecture diagram

Instance application scenario

4 Features and deployment of SIPGateway

Features of SIPGateway

Flexible incoming call mode

Full platform interoperability

MCU cascade

load balancing

High security

High availability

Data monitoring platform

Deployment mode of SIPGateway

5 Conclusion

Hot content

Popular comments of the whole site

About the author

Author's Album

Author's other popular articles

Hot News

Recommended attention

Hot software

OSCHINA Community

Online tools

Introduction

QQ group

Public account

Video number