Focus

 

• What is 3D face reconstruction technology? Essentially, 3D data is synthesized from 2D images, or 3D scanning is performed directly, and then facial repair is performed.
• Is the reconstruction expensive, and can ordinary people afford it? Most trauma and tumor related facial reconstruction operations are covered by medical insurance, and the public can benefit.
• How about the effect of 3D face reconstruction? It can basically maintain 95% recovery.
• What is the relationship between AI face changing, plastic surgery and 3D face reconstruction? AI face changing focuses on the realization of two-dimensional technology, while plastic surgery focuses on the application of 3D technology, and uses some advantages of 3D technology to achieve facial improvement.
• Will 3D face reconstruction technology be used in other fields in the future? There are cases of application of relevant technologies in the fields of mobile camera facial unlocking, digital human and so on.

 

summary

 

 

Lin Zhiying's car accident made people sigh with regret. After the accident, how to repair his face, which was damaged, became one of the topics of concern to the outside world.

 

The good news is that the hospital is evaluating the use of technology for 3D facial reconstruction to help him walk through the haze of the accident.

 

In medicine, 3D facial reconstruction is simply to reset the displaced bone segment of the fracture and reconstruct the face. First, 3D print the injured facial resin structure, simulate the restoration and reconstruction, restore the facial bone structure, and fix it with titanium plates.

 

It is understood that this technology has developed maturely abroad and has been widely used in the game and film industries. What is the point and mystery of 3D face reconstruction technology?

 

The following is the essence of live recording:

 

01

Get 3D face data first, and then repair the face

 

Liu Xingliang: After Lin Zhiying was involved in a car accident, it was said that his face was seriously damaged. Fortunately, the hospital news said that he could repair his face through 3D reconstruction technology. First of all, I would like to ask Mr. Wang to give us a scientific overview of what is 3D reconstruction technology, and what are the different types of this technology at present?
 

  *Parametric learning method based on 3DMM model

Wang Chengjie: From the perspective of computer vision, 3D reconstruction technology is essentially to effectively restore the 3D information of the face through the depth information of the face or 2D images, and finally complete the face repair. Generally, it can be divided into two types. The first is that it can obtain depth with some professional equipment, similar to 3D scanners. The other is to input one or more pictures and restore the 3D effect through some algorithms. There are two main ways to restore 3D from images, one is to use traditional 3DMM, and the other is to use neural rendering network for deep learning.

 

Liu Xingliang: What I understand is that one is 3D scanning, and the other is to take off the shelf pictures. What are the applications of 3D reconstruction technology in the medical field?

 

Zhang Jie: 3D reconstruction technology is widely used in medicine. For example, 3D reconstruction can be carried out for frequently applied CT examination. Generally, 3D reconstruction is carried out when fracture cracks or some lesions occur, so that the location and shape of lesions can be seen intuitively. In addition, when we are doing surgery design, we will print out a 3D resin model for experimental design and splice it. In particular, some tumor patients will have damaged appearance after tumor resection. We repair them and use image technology to observe the part that has not been removed, so as to evaluate the symmetry of both sides of the patient. Therefore, 3D technology is widely used in our field, and we are exposed to it every day.

 

Liu Xingliang: In addition to Tencent Youtu Lab, what other well-known enterprises on the market are studying 3D reconstruction technology, and where is the technical barrier?

 

Wang Chengjie: In fact, this 3D technology is applied in many scenes, so there are many companies engaged in this technology research. For example, we often contact the face scanning facial recognition in our daily life. In the process of login, in order to ensure the accuracy of recognition, the front camera of the phone has a 3D scanner. In addition, interactive app or video software also involves 3D reconstruction. For example, the headdress in Tencent conference is a composite effect, and 3D face reconstruction has been completed in this process.        

 

  *Application of 3D reconstruction in Tencent conference headwear function

 

In addition, the application of 3D technology is also related to film and television production. When we want to shape or change the image, in order to maintain the visual reality, we need to make adjustments from the perspective of 3D to obtain more natural effects.

 

02

If only bones are damaged, Lin Zhiying's appearance can be repaired

 

Liu Xingliang: Can Lin Zhiying's face really be restored through 3D reconstruction technology?

 

Zhang Jie: I don't think it's a problem, because I haven't seen specific photos. The serious facial fractures we see in daily life are mainly caused by two reasons: one is traffic injury, the other is high fall injury. There are mainly two aspects involved: the first is the fracture of bone tissue, because the facial bone tissue is particularly complex. There are maxilla, mandible, cheekbone, orbit and so on. Even if it is badly crushed, we will restore it. If the bone tissue is not damaged, we can basically achieve a more perfect anatomical reduction. Most of the applied incisions are also very hidden incisions, usually from the mouth, which will be more beautiful. The most important thing is the skin injury, because the skin injury and defect need to be recovered, and the recovery needs to replace it with new tissue, which will affect the appearance. If it is only bone tissue damage, I think it can be restored.

 

  *Lin Zhiying's daily photos of the injured scene and herself are from the Internet.

 

Wang Chengjie: From my point of view, this problem is a medical problem, because we do 3D reconstruction mainly focusing on facial surface reconstruction. Essentially, in our discipline, it does not involve internal bone tissue. In essence, what we say is to restore with 3D scanning equipment or images is to make skin surface. So from our point of view, we may just do something of reference value for the medical process.

 

Liu Xingliang: Can you give us a scientific overview of the process of 3D facial reconstruction? Whether the cost is high or not is within the range of our ordinary people.

 

Wang Chengjie: First of all, there are many methods for 3D reconstruction, but the current mainstream methods are basically machine learning. Then we can learn some original data to get the trained model. The latter model has a process from 2D images to 3D. So the current cutting-edge technology is actually a self supervised learning method. You can supervise yourself, from a 2D picture to 3D, and then from 3D to 2D. Then you can compare two 2D pictures. We call it the loss function in academia, and then train the model through this transmission method. In terms of cost, if high precision is required, there must be a R&D threshold. In fact, some senior researchers in our team are also trying to improve the effect of this technology. Now they can basically achieve the film and television level. It is mainly because the cost will be relatively large in the R&D stage, but not in the use stage. In terms of time, it is just a few seconds.

 

A previous article published at the CVPR conference in the field of computer vision was about the content of this field. The current algorithm framework supports the input of any number of images, one or two or three of which can be used. Generally speaking, the effect will be better if the number of pictures is more, but the input pictures should be in the same period as far as possible.

 

03

The cost is civilian, and medical insurance can be reimbursed

 

Liu Xingliang: Next, I want to ask Dr. Zhang, what is the current process of 3D facial reconstruction? How expensive is this operation? Can ordinary people consume?

 

Zhang Jie: 3D reconstruction technology is very common in medicine at present. For example, for fractures, we will observe the fracture site through CT of the patient's head, export the data to the company undertaking the 3D reconstruction task, help us print the resin model, and finally simulate the surgery process with reference to the resin model, and reassemble the fracture site to the original state. I think it is very helpful for surgery, especially complex fracture surgery or tumor repair and reconstruction surgery, which is very versatile, inexpensive, and absolutely affordable for ordinary people.

 

We now have a special team to carry out fracture surgery. In terms of tumors, we need to use other parts of the bones, such as fibula and iliac bone, to repair and reconstruct after tumor resection of facial maxilla or mandible, to restore the appearance and function of patients. The recovery of patients' function and quality of life after tumor resection is very important. For example, for abdominal surgery, gastrointestinal function should be mainly considered after incision. The appearance may not be so important, but the face is different. If the facial tissue is removed, the defect should be repaired, so that patients can have a good appearance and quality of life.

 

Liu Xingliang: How much does Lin Zhiying's operation cost?

 

Zhang Jie: Because the degree of fracture is not clear, the surgical cost is reasonably priced. As for surgery, small titanium plates are generally used to fix the broken ends of fractures, rivet the fracture parts to prevent them from loosening. After six months of fracture healing, we will take out the small titanium plates through surgery. The cost is divided according to the number of titanium plates used. The price of domestic and imported titanium plates is also different, depending on the materials. This kind of medical expenses are covered by medical insurance, and the price is completely affordable.

 

04

"Change a 20-year-old face" depends on plastic surgery

 

Liu Xingliang: How about our 3D reconstruction effect? How much can Lin Zhiying possibly recover?

 

Wang Chengjie: From the technical point of view, we will do quantitative evaluation. If the reconstruction is carried out, the average error after reconstruction and the error of the real surface should reach 1 mm to 2 mm. Of course, this is an academic indicator. In fact, ordinary people may be more concerned about whether they can recognize the reconstruction model by observing the 3D grid shape after reconstruction. We have done a round of experiments internally before, and let people guess who the reconstructed character is by reconstructing the 3D model, and found that there is a 70% or 80% probability that we can guess correctly. Of course, this also has something to do with appearance and appearance, because some people look closer to the average face, so it is relatively difficult to guess. Lin Zhiying's face is also quite distinctive, and most of it can be recognized after reconstruction.

 

Zhang Jie: I think from the medical point of view, we will first reset him and build up the contour, but the more important thing is the recovery of soft tissue, because there are skin muscles under the face and so on, especially the attachment of facial expression muscles in the repair process. Although the final repair is completed, the effect may not be particularly good. However, 95% recovery should be no problem, but there may be many differences between small expressions or individual fine parts and preoperative ones.

 

Liu Xingliang: 3D reconstruction technology We can use pictures for 3D modeling. If I am Lin Zhiying, can I choose to use specific pictures to restore to my twenties and thirties? Is there any technical obstacle to this undertaking?

 

Wang Chengjie: Because the current technology is essentially to receive input pictures, as long as the pictures at a certain time point are complete, 3D restoration can be performed based on the pictures. For some films and TV plays, the whole story line may be relatively long, with stories of 20 years old and 40 years old. Our technology is to realize the migration from the current state back to the age of 20. If the complexity is not in the same dimension compared with the medical perspective, AI technology is still simpler from the perspective of scenario differentiation.

 

Zhang Jie: This is a big problem. In addition to the level, it also involves the concept of anti-aging. If you want to recover to 20 years ago, although the bones will no longer develop after adulthood, the main changes are soft tissues, which are affected by gravity, collagen loss, skin sagging, wrinkles, etc. It does not mean that you can recover to 20 years old when you want to recover to 20 years old, which requires a lot of money.

 

Liu Xingliang: In the film and television works, even though Lin Zhiying has a gap after facial repair, is there any problem with the restoration based on the later technology?

 

Wang Chengjie: This kind of technology is called content generation technology. It has the ability to learn by learning the state of an actor at the age of 20, including facial structure and texture. After completion, it can be transferred to the current film and television works, which is visually equivalent to returning to the original appearance.

 

05

Relationship between AI face changing, plastic surgery and 3D reconstruction

Liu Xingliang: Now some movies and TV plays will use AI face changing to replace roles. Is this technology also used? What's the difference between plastic surgery and 3D reconstruction? Can we draw an equal sign between the two?

 

Wang Chengjie: There is still a difference between AI face changing and 3D face reconstruction. Now ordinary AI face changing can still see obvious defects or effects, which is not very natural. Sometimes facial features change, but the contour does not change.

 

In the field of film and television, the requirements of face related processing technology are relatively high, which requires the cooperation of technology and people.

 

With the development of current technology, 3D reconstruction related technologies become more mature, which can help film makers to use their imagination. With these technologies, they can break through age changes and other problems, so as to focus on creativity.

 

Zhang Jie: Plastic surgery is a major category of maxillofacial surgery, which is also a very important department in our hospital.

 

The improvement of people's living conditions, such as the ground covering the sky or the small chin, as well as tumors or facial deviation caused by diseases, are all conditional on plastic surgery to improve. The 3D design model is used to change the patient's appearance, make it conform to the public's aesthetic, and enhance the patient's social identity and self-confidence. This kind of surgery requires the use of 3D technology. Now this kind of surgery is very popular, and it is carried out at its own expense.

 

06

3D reconstruction has been widely used in mobile phones, digital people and other fields

 

Liu Xingliang: Today's 3D reconstruction technology is in the medical field. What other fields can be applied except medicine? Or you can share some classic application cases with us.

 

Wang Chengjie: There are many application scenarios of 3D technology. In fact, these scenarios will be designed to be similar to entertainment interactive 3D applications. Of course, we focus on film and television level 3D technology. We can also help artists "how to make a fat person thin" with 3D reconstruction. Future algorithms can also automatically realize more possibilities in film and television works. 3D is often hidden inside, providing support for other explicit technologies.

 

For example, the face recognition unlocking function of mobile phones is also realized through 3D scanning equipment or reconstruction technology. There are 3D models in the mobile phone. When you unlock the phone, the two models will be compared to see if they are the same person. When we set face unlocking, we will turn our head. The process of turning is actually 3D reconstruction.

  *Comparison of LAP 3D facial reconstruction (far right) effect and other facial processing details

 

We now have many technologies, which may be on the 2D level. If 3D information is added, the effect will be significantly better. For example, when we do facial beautification or makeup, it can be done in 2D. If we introduce 3D reconstruction information, it will be better. With the help of 3D reconstruction technology, the actors will be given makeup, and 3D will appear more natural.

 

In addition, there are more digital people in the whole industry. Its digital virtual process is actually the function of 3D technology. For example, if we want digital people to do some facial expressions or lip movements, there are 3D reconstruction and 3D driven technologies behind them. If we want digital people to be more authentic and have more natural conversations, it can not be separated from the improvement of 3D technology.

 

Zhang Jie: Is this figure useful?

 

Wang Chengjie: In many of our industries, the cost of customer service receptionists is high. The customer service handles the reception through the network. In this process, there are real customer service people who communicate with users through video or voice. The problem is that the labor cost will be high and the time will be longer. However, digital people can achieve this kind of experience of communicating with real people, and there will be no resource waiting or resource shortage.

 

Liu Xingliang: So what else might we use this 3D reconstruction technology in the medical field?

 

Zhang Jie: 3D printing. Repair human bone defects with 3D printed hard tissue. When encountering irregular repair parts, we need to directly repair them with 3D metal printed prostheses after design. The finished product is very standard and designed according to the image. Now 3D printing of hip and cervical vertebrae is also being done in medicine, gradually realizing personalized development, which is also the general trend of 3D printing. In addition, the hospital also has AR and VR technologies for real-time assistance during surgery to meet the technical requirements of computers.

 

Liu Xingliang: If two teachers cooperate in the future, what support can they provide to each other?

 

Zhang Jie: In the aspect of artificial intelligence learning, we have also carried out many research topics, including intelligent sketching technology or 3D photography to evaluate the degree of surgery, which can help us accumulate experience in the later stage.

 

  *AI generation content scheme based on research generative countermeasure network GAN and 3D technology

 

Wang Chengjie: AI can accomplish many tasks, and the popular concept is called counter generation network, which can accomplish generative tasks. For example, there is a picture in the film and television field that has content vacancy due to occlusion. The algorithm technology can make up for the vacancy through AI's understanding of the scene.

 

As mentioned before, the machine may be able to automatically generate missing parts based on past experience in the future. Tencent Youtu Lab has a machine learning platform, Tione, which can solve some problems in various industries through platform learning. We are also trying to generalize or platform AI capabilities to all industries. For example, in the field of medical CT, there will be more detailed information input. From the perspective of AI model learning, it will also be easier. I believe that AI can achieve the technical and functional support that Zhang Jie hopes in the future.