Before the system went online, Alibaba Cloud service teams went deep into customer business scenarios, improved the observability and cloud product monitoring under the cloud native architecture, and accessed full applications and corresponding business logs to ARMS and SLS, significantly improving the ability to actively detect and white box locate distributed system errors.

Through PTS products and combined with actual business scenarios, the full link performance of 11 core business scenarios was pressure tested and optimized. Two major risks were released in advance and eliminated. In combination with the pressure test results and best practices, the resources of more than 30 products and hundreds of pods in the whole link were comprehensively patrolled and optimized. At the same time, the capacity of Hologres, PostgreSQL and other instances was expanded, Through the optimization of more than 200 SQL statements, the risk of a large number of slow SQL problems has been timely converged.

During the launch of the system, Alibaba Cloud service teams were stationed throughout the whole process, responding to and dealing with the urgent problems of customers when they went online as soon as possible, and quickly handled two unexpected exceptions. First, the database connection pool is not well controlled, The number of concurrent connections of some work nodes in Hologres is full, leading to data migration failure and abnormal application startup. By quickly locating applications with soaring connections on site, and optimizing the free link release parameters on Hologres side, customers can quickly avoid problems. Second, the SQL diagnosis optimization function of the PostgreSQL database triggered the defect of the index_advisor plug-in, which led to the abnormal restart of the corresponding PostgreSQL instance. The customer was urgently asked to stop using the SQL diagnosis optimization function, and the priority was to ensure that the database was stable and the system was normal.

The OMS The containerization and distributed transformation of the three core systems of POS and CRM, and the success of one-time online cutover, accelerated the process of digital intelligence strategy.

The Alibaba Cloud service team, based on the customer's business characteristics, combined with the best practices of Alibaba Cloud's microservice governance and stability governance system and customer cases, helped customers develop plans for application lossless publishing, grayscale publishing, capacity planning, flow limiting and degradation, and worked with customers to complete the implementation of the plans.

Lossless application publishing can effectively avoid the problem of front-end exceptions during application publishing, improve the user experience, and make the system more smooth when changing. In order to speed up the landing and reduce the workload, make full use of the ability of ACK and MSE, realize the elegant exit of applications by configuring preStop and MSE to go offline without loss, protect the safe start of applications by configuring health checks and enabling MSE online traffic preheating function, and effectively avoid the traffic loss caused by application release. Customize cloud efficiency pipeline for customers based on their current situation The application release scheme combining ACK and MSE microservice governance capabilities can realize lossless online and offline in the process of application release, without adding additional operations, which reduces the burden on technicians.

Gray publishing can roll back fast stream cutting in time when major exceptions occur after the application goes online, and can try and error the new version with small traffic. Using MSE to provide agent access, a field in the header based on HTTP is colored for traffic, and Kubernetes' declarative deployment is used to insert a grayscale identifier into the application version, so that only colored traffic can enter the version marked with a grayscale identifier, realizing the full link grayscale capability based on logical isolation mechanism, This enables small-scale traffic verification of the business after the release of the new version. Once any problem is found in the new version, it can be rolled back in time to minimize the impact on the business.

Full link pressure testing is the best way to find system bottlenecks and determine system capacity. The customer technical team and Alibaba Cloud service team sorted out more than 50 pressure testing scenarios for the five core systems on the cloud, and designed a full link pressure testing scheme, data model and script. Perform optimization, table partitioning key, shard optimization, index adjustment, and application log asynchronization through Hologres SQL, JVM parameter optimization, code optimization, Redis cache, More than a dozen tuning measures, such as MQ architecture transformation, have improved the system stability and overall system performance by more than 50 times, significantly improved the utilization of resources on the cloud, and reduced the overall resource cost by more than 5%.

The current limiting and degrading capability can install a reliable safety rope for the system. Since there are many slow SQL and time is too short to optimize, in order to avoid serious slow SQL that will drag down the entire database and cause interruption risk to online business, Bosiden has used MSE's database governance capability to evaluate the peak flow based on the pressure test results, configure SQL flow restriction rules, and selectively let SQL wait or fail quickly when the database traffic is too large, Turn uncertain traffic into deterministic traffic to ensure the stability of the database and the stability of the entire business.