Solution case of 100% ECS CPU utilization and high system load

When Server CPU utilization rate 100% When the system load is too high, you may suspect that it is infected with viruses, hacked, or attacked, or there are problems with the website building program, java program, and test program. If such problems still occur frequently after you redo the system, there is another possibility that you have not noticed. It is from the official monitoring program.

Lao Wei suggested that there are many reasons for 100% of ECS CPU usage and high system load. This article shares a real case, which only represents one possible reason and solution. This kind of problem is not all caused by Tencent Cloud Cloud Mirror. Different server deployment programs, different visits, low host configuration, different security protection, and different managers' technical level may lead to 100% CPU utilization. What Lao Wei encountered is an example, which only represents the individual situation of the caller. It cannot be said that 100% CPU load is the reason!

100% CPU utilization and high server load

A friend asked for help and said that the server CPU load was 100% and the server load was too high, which happened almost every few days. It was a headache. In order to reinstall the server operating system, he fell ill again soon afterwards. Log in to Tencent Cloud Console After carefully observing the monitoring details, it is found that the time is mainly at 0:00 midnight, as shown in the following figure:

If the monitoring time is shortened a little, it will be seen that at 0:00 every few days, the CPU will be 100% full in a short time (lasting about 10 seconds), and the bandwidth, hard disk I/O reading and writing will also change significantly.

Maybe it's the cloud mirror program

With the help of Lao Wei, he ruled out many possible problems such as the site building program and the pagoda panel. Later, Lao Wei found that a planned task was executed regularly at 0:00 every day through the professional version of the pagoda panel's Task Manager>Scheduled Tasks. The program name has the words YunJing and YDCrontab, which is Tencent Cloud's cloud mirror monitoring program.

It is learned that this program belongs to the item checked by default when buying the server. Cloud Mirror is also called host security. It is a security protection service provided by Tencent Cloud, such as hacker intrusion detection and vulnerability monitoring. Alibaba Cloud also has a similar cloud security server called Yundun (Anqi). Both Yunjing and Yundun will regularly and irregularly start security scanning to check your server for security information.

Find a way to solve the problem

It is speculated that the program startup scanning at 0:00 every day may cause great pressure on the server CPU, load, bandwidth, and hard disk. Then think about how to solve it.

After asking Tencent Cloud customer service, we learned that there are two solutions:

1. Upgrade Cloud Mirror

View the cloud mirror version by executing the command

/usr/local/qcloud/YunJing/YDEyes/YDService -v

If the version is low, try upgrading to the latest version. For example, the command to upgrade to version 2.1 is as follows

wget http://u.yd.tencentyun.com/ydeyes/download/ydeyes_linux64_2.1.0.198.tar.gz && tar zxvf ydeyes_linux64_2.1.0.198.tar.gz && sh self_cloud_install_linux64.sh

For the latest version of Cloud Mirror when you encounter such problems, you can issue a work order to ask customer service for it.

Continue to observe after upgrading. If the CPU is still continuously full, please see the solution below.

2. Uninstall Cloud Mirror

Specifically located in Tencent Cloud Console >Host Security>Host List>Find the right side of the server and click Uninstall, as shown in the following figure:

Click [Uninstall] and wait a few minutes before refreshing the page to see if it has changed to "Reinstall" and if the status shows "Offline". If it remains unchanged after a few minutes, execute the following command:

/usr/local/qcloud/stargate/admin/uninstall.sh
/usr/local/qcloud/YunJing/uninst.sh
/usr/local/qcloud/monitor/barad/admin/uninstall.sh

Alibaba Cloud's Linux server executes the command:

/etc/init.d/aegis uninstall

You can uninstall Yundun.

Then observe the server. If the data is stable for a long time, it will be OK.

matters needing attention

1. When you buy a server, you can uncheck the program of host security (cloud mirror) to prevent it from appearing in the server; If you encounter this situation in use, consider uninstalling.

2. After uninstalling Yunjing and Yundun, you can no longer monitor the server details, such as CPU, memory, hard disk and bandwidth. If there is a pagoda panel, you can see relevant data in [Monitor]. These three cloud products are all displayed by calling the system's underlying interface data, and there is no difference.

more Click me directly According to the official website of the pagoda panel, the pagoda will officially present a coupon package of 3188 yuan. New users can enjoy a professional version of 0.99 yuan experience for one month.

3. Some servers do not cause so much pressure when they start scanning after installing cloud mirror, and normal business access will not be affected. Even if the impact is very short, you can make plans by observing the scanning time and monitoring results of the monitoring program on your server.

After nearly a month's observation, the CPU load rarely reaches 100%. Basically, this problem has been solved. Finally, Lao Wei reminded everyone to analyze the specific problems and prescribe the right medicine.

Generally speaking ECS CPU utilization rate 100% And high system load, It's not all due to the surveillance program scanning 。 For example, this happens when the server configuration cannot meet the current business needs. Therefore, when encountering such problems, we should first use the exclusion method to eliminate them one by one, then look for clues, find out the culprit that causes the full load of servers, and then solve the problem.