Nowadays, when people talk about public cloud, they have habitually put the cap of serving small and micro enterprises on public cloud, and most cloud service providers will face the following doubts: 1. Whether the public cloud can really save costs when the size of the start-up company increases to a certain extent. 2. If mature products are put on the public cloud, can the current domestic cloud service providers be reliable and provide enough stability. The author thinks that all domestic cloud service providers are holding their breath in silence, and must rely on their services and mature products of a certain scale to justify the name of the public cloud. As a cloud service provider established for three years, Qiniu already has a number of mature products with massive data. It also wants to share experience and cases in this regard, hoping to give some reference to mature products that try cloud services for the first time.
To cloud, loved, hurt?
I have encountered many Internet products with massive data. At the beginning, I thought that all services should be in my own hands, which is the most controllable. Some of them also tried some cloud services, but if the service quality did not give them enough confidence at that time, then for a long time, most of their services were self built. For example, the image origin storage chose to use some open source tools in the industry.
However, in the past two years, with the prevalence of lean entrepreneurship concept and the further improvement of cloud services, they will start to reconsider the use of third-party services. The concept is gradually changing into: professional things, let more professional people do it, and the team is more focused on business optimization. It is called "good and false in things". two thousand and fourteen The mobile social platform Momo, which was listed on NASDAQ in, went through such a process. two thousand and fourteen In, when the number of self built source site storage users exceeded 100 million and the number of monthly active users increased to tens of millions, they came up with the idea of Qiniu Cloud when they encountered the pressure of access.
Try "Ox Knife"
Qiniu platform has hosted a batch of mature Internet products with massive data, including many listed companies, many of which are UGC Class products, such as Hikvision, sing it, camera360 , pea pods, visual China, etc. The experience here is that mature products will conduct very systematic and cautious function and performance tests when choosing a third-party cloud service due to the suspicion and cautious attitude they have held on cloud services before. This test mainly includes:
1. First is the function. See the provided API Whether the customer's needs for scenarios can be fully considered, and whether sufficient flexibility is left for the caller. In addition, the user of the function is the programmer, and API It's the programmer's UI (user interface). API It should be friendly to programmers, simple, intuitive, easy to use and elegant.
In the case of Momo testing Qiniu, according to the characteristics of the product business operation, they tested from different size file upload, download, thumbnail processing, custom domain name binding, and Qiniu's unique image storage and other function points.
2. abundant API It is easy to use and can be accessed quickly to meet the characteristics of the scene. The next step is to test the stability of the source station.
Testing stability usually involves putting a small part of the measurement run for a period of time to see various data. In order to test the stability, Momo lost 1/10000 of the amount to Qiniu and ran for a week. When a mature product conducts a preliminary stability test on Qiniu, it generally uses the "image storage" function of Qiniu. Image storage refers to the use of additional 7N as the image server when the data is stored at the customer's source station. For customers who already have massive data, this function can easily feel the acceleration and "security". Through the developer background of Qiniu, you can set image storage for the origin site, so that users of the website can read from Qiniu cloud storage every time they visit. If there is no data on the cloud that users need to read, the 7N server will pull resources from the source station and then return them to the end user. Inside the Seven Bulls, this function is also called "second to second", which means you can feel it.
The three views are correct, from one ten thousandth to ten percent
In the process of testing, the service provider and the customer will also have very in-depth exchanges in terms of technology selection, team, products and service concepts, and the customer will also evaluate cloud services from the "three perspectives". The three views are correct, which will greatly promote the follow-up cooperation and provide a good foundation for mutual trust. In this era of service as the king, every cloud provider should follow the service concept of focus, professionalism and perfection.
In the process of Qiniu promoting cooperation with mature products, another technical detail was taken into account by the customer: Qiniu's service terminal uses Go Written in language, Go Genetic support for parallel computing is very suitable for building cloud service systems, so many customers also approve of this model selection.
Back to the case of Momo, after a series of tests, the results were very satisfactory to Momo. However, since it was already a mature Internet product, it was decided to be responsible for the user experience after the test 10% Measure it to seven oxen.
Professional and perfect
Social products are different from e-commerce or other products. For social applications with over 100 million users and tens of millions of active users every day, the number of back-end system files can reach 10 billion. In the overall visit, the traffic of pictures accounts for 90% Above, and because it is accessed on the mobile device, it is less than 8k There are so many pictures, and the display quality, upload time and access speed of pictures are key factors that directly affect the user experience. In addition, for social UGC, The technical difficulty lies in that no data is a dead end. All data are theoretically "hot data", and the traffic is scattered, but the overall access pressure is very high. In addition, the sudden hot spots in social networks cause a sharp increase in concurrent access to a single file, which will also pose a huge challenge to basic services.
This feature is based on open source technology for storage. With the growing business pressure of the origin site, it is difficult to achieve accurate storage bug location. These problems are commonly encountered by many mature products with self built storage.
Third party cloud storage services that have served multiple similar products and gone through various pits will have more experience in these aspects. Here's how Seven Bulls solved the problem:
1. Massive storage.
A wide area network distributed architecture with multiple data centers is adopted to build several core storage rooms nationwide. Customers can actively choose a storage room according to their own business characteristics or completely hand it over to the cloud service system to schedule data. In addition, mutual backup technology is used between the two data centers to interconnect the two data centers with bare optical fibers. When users upload to a data center, the system asynchronously synchronizes the file data and related original data to the other data center, so that when a data center fails, the storage service will enable different emergency plans according to the level of failure, Switch the request to the alternate data center.
Technically, the core demands of the storage system are cost and reliability, which are also a pair of contradictions. To reduce the risk of data loss, it is inevitable to increase the number of copies of each data copy, while increasing the number of copies of each data copy is bound to increase the cost. To solve this problem, Qiniu used EC Redundancy algorithm can balance the contradiction to achieve low cost and high availability.
2. Response to flow pressure
As a cloud service provider, to carry more mature services, it is very important to refine the operation. You can consider the characteristics of different types of customer services, and leave enough flexibility and scalability when designing cache and system logic. Cloud services need to establish a hierarchical storage mechanism and migrate according to object access hotspots. Unify the distribution system of internal data flow, and manage the processing flow of message flow uniformly, so as to have enough capacity to deal with peak requests of different customers and make the service more stable. At the same time, the response time is monitored in real time to ensure that problems can be perceived and handled as soon as they occur.
3. Powerful pictures / Audio and video processing service.
In the era of scenario, more and more data of vision and rich media will be generated. Some of them are generated actively by users, and some of them are generated passively, such as camera data. It can be predicted that the cloud storage platform will carry a large number of image, audio and video data, of which the real-time response ability of cloud storage services to image processing is also the key. Take Momo for example, half of the images in every second need to be processed, that is, close to five thousand The number of times of real-time image processing should be about, 99.9% They are all processed within 100 milliseconds. In addition, the original JPG Format to WebP Format, which can achieve better compression ratio under the condition of the same quality and save costs.
The same is true for real-time response to video. At present, for example, for short video applications, there are some other game video sharing, that is, after users record on their mobile phones, they transmit them in the background ten You can share in seconds. After sharing, users of other terminal devices will access it. The format of access is not necessarily the same. If the customer adopts self built, this kind of real-time requirement is high. It is a long way from the back end hundreds of kilometers, thousands of kilometers of machine rooms, through various routers, to the user community's LAN, and the environment is difficult to control. Then to the real server, it must have fast data and strong processing capacity. Then there is high throughput, and the number of users is large. One server can only support hundreds of users a second KPS In this case, it is still a good technical difficulty for customers to build their own response to ensure performance. From our observation, the server processing should not exceed one second, and the user experience will be greatly affected if it exceeds one second.
4. Add user-defined processing to meet the special data processing needs of some customers. This is a characteristic service of Qiniu. As a general framework, the processing services provided by cloud services may not include the non general data processing requirements proposed by some customers. Therefore, for this special requirement, customers can define their own data processing logic. The customer said that it was OK to buy a processor in our computer room. However, the next step is to cover all operations and machines. This process is very tiring. Custom processing allows users to package custom programs and run them in this computing framework. After the user's application is processed, it enters the specified framework, allowing the user to experience their own logic by plugging in. This service can also enable more developers who deal with unstructured data, such as voice recognition and image recognition service providers, to join the Qiniu platform to provide processing capabilities to our customers, so that we can work together to build an ecosystem.
5. Professional service awareness. It is needless to say that service awareness is of great importance to cloud service providers. To quickly locate and solve customer problems and respond to customer needs is one of the criteria for a good cloud service provider.
Fully embrace cloud storage
With the growing recognition of Qiniu's technology and services, a few months later, Momo made a decision to officially take Qiniu as the provider of origin station and off-site backup, and transfer all image processing to Qiniu. It is also Momo, which has always maintained its entrepreneurial spirit. It needs a company that is sustainable, stable, promising and has technological feelings to provide stable and high-quality cloud services. Seven Bulls just met their expectations.
The first step to fully embrace the cloud is to migrate historical data to the cloud. This is a big project, but it is an extremely critical service step for "mature products using cloud services". First, the following problems may be encountered during the migration process:
1. Data error occurred during network transmission during hard disk migration.
2. Rate problem during transmission. Take Momo for example. If common transmission methods are used, the entire migration process will take half a year and will seriously affect online business.
3. The hard disk is damaged during sending and using.
4. Store the data file to merge the database. If the traditional database writing method is adopted, it will take several days, which will bring unnecessary huge pressure to the customer database and cloud service database.
For these problems, a comprehensive migration system is particularly important:
1. Customize the data format and import the stored data according to the customer's business data model.
2. The database structure needs to have strong expansibility, which can realize instantaneous consolidation.
3. The migration system takes full account of uncontrollable factors, such as hard disk damage and power failure.
So far, Qiniu has become a very good partner of Momo, supporting their data storage, access acceleration and image processing of nearly 200 million users. Cloud services help small partners Momo solve the technical pressure brought by the subsequent business growth scale, so that they can pay more attention to their core business.
You may say that the domestic cloud ecosystem is not yet mature, but after several years of baptism and countless pits, the current domestic cloud service providers are becoming more and more diligent and professional. We also see that more and more mature enterprises, like start-ups, choose to trust professional third-party cloud services, so they can focus more on their own businesses. two thousand and fifteen It can be predicted that the cloud will be more exciting.
