Reviewing the development history of mobile phone imaging system, is self-developed chip the ultimate goal?

mobile phone What is the end of the imaging system?

The mobile phone image system has gradually become the focus of mobile phone manufacturers. From being photographed to being photographed well, these seemingly simple words need a lot of efforts from mobile phone manufacturers. In order to meet the increasing demand of consumers for mobile imaging, mobile phone manufacturers are also trying to change.

In recent years, mobile phone photography is both convenient and fast, with bright colors and excellent optimization. In the past 2021, mobile phone imaging system has achieved many breakthroughs. Next, I will take you to make an inventory of the major events that have occurred in the mobile imaging system in recent years.

01 The image reform gradually slows down

In the early development of mobile phone imaging system, hardware has not become an important driving force for manufacturers. Taking photos only meets the basic needs of users. At that time, the big bottom, high pixels Micro cloud Taiwan has not really entered into our life. Photography with mobile phones is just a joke 。

When the functions of mobile phones become more and more abundant, the image ability ushers in a real change. In order to bring users more powerful image capabilities, the rear multi camera scheme has been introduced to the market. Compared with single lens, multi camera has obvious advantages in background virtualization, zoom and night scene imaging. The emergence of multiple cameras marks that the image ability has become the next function point pursued by mobile phone manufacturers.

With the continuous strengthening of the image system, stacking hardware has become an important driving force for mobile phone manufacturers, but there will always be physical limits to hardware. When the mobile phone image module becomes larger, it will affect the weight and feel of the mobile phone to some extent, such as Xiaomi 11 Ultra 、 Huawei P40 Pro+ For example, although they are highly praised for their imaging ability, it is difficult to control their weight within an ideal range. In order to get a better photo experience, it goes against the original intention of portability.

Due to the limitation of internal layout, the space reserved for camera module by mobile phone is very limited. How to improve the image quality of the main camera as much as possible without increasing the size of the rear image module has become an important driving force this year. After a lot of market research and user feedback, we found that the use rate of periscope telephoto lens is not high, and it naturally faded out of everyone's view as a medium-term transition product of the image system.

02   Software "Volume" becomes the first battlefield

How to evaluate the quality of photos? In addition to being able to take pictures, it is more important to see the detail performance, color performance and image processing ability. Nowadays, many mobile phone manufacturers have gradually shifted their focus from the hardware arena to the software arena, realizing "one time imaging" with the support of self-developed image chips and more new technologies. For example, the computational optics technology used in Huawei P50 Glory Magic3 Multi subject fusion technology and other brands of computational photography. In fact, these terms all have a common feature, that is, they are more powerful and can make photos more beautiful through post optimization.

We have just listed Xiaomi 12 Series as an example. Xiaomi 12 Pro The rear 50MP ultra clear three main shots are adopted, namely, 50MP wide-angle main shot, 50MP portrait main shot, and 50MP ultra wide-angle main shot; and Xiaomi 11 Pro 50MP main camera, 8MP periscope telephoto and 13MP ultra wide angle are used. Xiaomi 12 Pro Xiaomi 11 Pro cancels the 8MP periscope telephoto and adds the 50MP portrait camera. Although it is more convenient for portrait creation, the loss of the periscope telephoto also greatly reduces its magnification. In addition, the GN2 1/1.12 "outsole used in Xiaomi 11 Pro was replaced with an IMX 707 1/1.28" sensor.

In order to balance the image capability, the Xiaomi 12 Pro is equipped with a new "Xiaomi Image Brain", which includes four functional modules: intention recognition unit, acceleration engine, computing unit and ecological engine. Compared with the old model, the photography speed is increased by 53%, the continuous photography speed is increased by 209%, the night photography speed is increased by 40%, and the cold start speed is increased by 21%.

Through Xiaomi 12 Pro, we can easily see that Xiaomi has slowed down the pace of the mobile phone video system. The castration image system focuses on multi camera technology to ensure image quality. However, there are some differences between multi subject photography and multi subject fusion computational photography. Next, let's talk about the latter.

The original idea of multi subject fusion computational photography is that multiple cameras can "communicate with each other" through algorithms, so as to achieve the effect of 1+1>2. Multi subject fusion computational photography started from the Glory Magic 3 series. In brief, multi subject fusion computational photography can be understood as full lens participation in full focus fusion technology.

In the wide-angle focus section, the fusion of color main camera and ultra wide-angle camera is adopted; In one time shooting, color main camera and black and white camera fusion are used; In the focus section below the telephoto, the color main camera and telephoto fusion strategy are used intelligently according to the effect, while the telephoto lens uses multi frame fusion. The main advantages are as follows: first, break through the limitations of sensors and use computing power to exchange image quality; Secondly, the subjects are clearer when shooting distant scenes; Finally, it supports heterogeneous chip platforms, not subject to chip constraints, and unifies image styles across chip platforms.

Obviously, with the continuous improvement of people's image ability, it is very important to have strong computing power or not. As a pioneer in the field of mobile imaging, Huawei also has excellent optimization solutions.

P50 series adopts a new optical module, whose primary purpose is to reduce weight and volume to meet the balance of grip feel.

In brief, computational optics can restore the loss in optical signal transmission to the greatest extent by changing the "modulation transfer function (MTF)". Previously, image processing without computational optics was called half segment image processing. The front optical part lost 50%. Later, we tried to restore and restore, and only 60% of the original image information could be restored.

On P50 series mobile phones, 25% more image information can be restored due to the addition of computational optics. 60% of the image information can be restored from the previous half stage, which can be increased to 81%. Without taking up too much mobile phone space, the imaging quality brought by high-performance lens module can also be obtained, which is worth learning for manufacturers.

Through the inventory of several imaging technologies, we believe that we have a deeper understanding of optimization. In fact, such picture optimization schemes have appeared for many years Huawei P40 The XD Fusion computing photography engine has been launched for the first time in the XD Fusion Pro series, and has now been upgraded to XD Fusion Pro. To put it simply, optimization is not only the driving force of mobile phone image system in 2021, but also the new upsurge of mobile phone image in recent years.

03 Chip will become the second battlefield

Software optimization is very dependent on the computing power of CPU and GPU, and powerful performance reserves become very important. If you want to have a better photo experience, software alone is still pale. In order to meet everyone's high requirements for optimization, the hardware should also keep up.

I have to say that Vivo's response to the image hardware is very rapid. With the gradual popularization of super wide angle and multi subject photography, photography is breaking people's imagination of mobile photography optics. In the face of complex light, dark scenes, extreme night scenes and many video shooting scenes, the image computing power and chip power consumption of mobile phones need to be further upgraded and evolved. So Vivo's first independently developed professional image chip was born vivo V1 。

Left: without V1 chip Right: with V1 chip

Compared with the design of the whole SoC chip, the threshold and cost of the self-developed image chip are more suitable for rapid iteration. It does not require a particularly advanced process, but also has a certain fault tolerance. In the actual experience, the models with vivo V1 chip are more excellent in night view and portrait.

At the same time, OPPO also released its first independent chip, Mariana X, at the end of the 21st year. This chip adopts the 6nm full EUV process and is in the same echelon with many mainstream smart phones 5G SoC. It was mentioned in the press conference that Mariana X can achieve 18TOPS AI peak computing power stronger than the 15TOPS computing power of Apple A15 processor NPU unit.

As a more important standard to measure the performance of mobile phone NPU - computing power per watt, Mariana X is 11.6TOPS/W, which translates into the peak computing power consumption of the entire chip to about 1.5W. If it is allowed to perform the processing of real pictures, that is, actually captured pictures, it will be up to 12TOPS/W, and its performance and energy efficiency ratio are at the leading level in the current industry.

In addition, Mariana X also has powerful data throughput and AI computing capability, which can be combined with customized RGBW sensors for RGBW Pro mode enhancement processing. Although Mariana X has not been installed on the mass production model, its real strength can be seen from the data and process technology.

The author has something to say:

The development of mobile phone imaging system has gone through many evolutions so far, and each change can bring a new experience. In 2021, the focus of mobile phone manufacturers has gradually shifted from the hardware market to software optimization, from "taking clear pictures" to "creating good photos"; From simple pixel improvement to picture optimization, this seems to be a simple change, but in fact it is the full link of mobile phone images. So in the next year, you will see more models based on optimization. Maybe in the near future, mobile phones will also produce blockbusters.

This article is an original article. If it is reproduced, please indicate the source: Reviewing the development history of mobile phone imaging system, is self-developed chip the ultimate goal? https://mobile.zol.com.cn/785/7856374.html