SEO spider log analysis helper: Spider Analyzer plug-in introduction and download

We are doing it for the website SEO optimization One of the most important tasks is to analyze the spider crawling. If the spider doesn't crawl your website, it will not be included. How to observe the spider crawl log? Daddy recommends the WordPress plug-in of Spider Analyzer, which is simple and easy to use.

What is a spider?

The spiders mentioned in this article refer to the crawler robots of various search engines. Because they will crawl one by one along the links of your website, we all call them search engine spiders.

Web crawlers (also called web spiders and web robots, more often called web chasers in the FOAF community) are programs or scripts that automatically grab information from the World Wide Web according to certain rules. Other names that are not often used include ants, automatic indexing, simulation programs, or worms.

Why observe spider records?

The Internet is full of various web crawlers, including search engine spiders, such as Baidu, Bing, Google and other bot robots, as well as many garbage robot crawlers, such as MJ12Bot, AhrifsBot, MauiBot and so on.

Garbage crawlers will occupy additional server resources. Some small servers will be hung up by garbage crawlers if they can't do a good job, making the website inaccessible. Therefore, we need to constantly observe whether the website has been crawled by garbage crawlers, and also need to pay attention to whether the search engine crawlers have crawled pages in the past, and how often, so as to make targeted SEO adjustments.

Introduction to Spider Analyzer plug-in

Spider Analyzer is used to track spider crawling logs of various search engines on WordPress website, and conduct detailed spider crawling data statistics, spider behavior analysis, spider crawling analysis and pseudo spider interception.

This plug-in is divided into two versions, the free version and the Pro version. You can search and install the free version in the WordPress background plug-in library directly. Some functions demonstrated in this article are only available in the Pro version.

In Spider Statistics, we can see which search engines or web crawlers have the most times to crawl your website.

As can be seen from the above figure, the number of times Google grabs my dad's website building notes every day is the largest, and the second place is MauiBot. The search found that it is a garbage crawler, which will be directly blocked by him later, wasting server resources and bringing no traffic.

From here, you can see which URLs on the website are the most crawled by spiders. You can properly insert an inner chain on these pages for some newly published articles or articles that want to increase the weight, and guide spiders to crawl and crawl.

Similarly, popular articles can also be inserted into other articles for spider guidance.

Then you can also check the included status. If you find an article that is not included, you can check the quality of the article and submit it to the search engine for grabbing.

Click the icon behind the number of web addresses to view the specific spider crawling. For example, in the above figure, we can see that this spider named coccobot crawls the site map of the website.

Then my father searched online. This search engine is Vietnamese. For our Chinese website, the capture of Vietnamese search engine is meaningless, so it can be directly intercepted. This interception is a direct interception on the server, rather than using robots. txt. It is simple, crude and effective.

In the spider list, we can see the spider list, spider IP segment, suspected fake spiders, and spider interception.

The SemrushBot in the above figure is a robot of the famous SEO toolmaker. If you don't want people to analyze your website too thoroughly, you can directly block its crawling. It is not easy to judge whether the IP is really a disguised robot. Although it is Shanghai IP according to the IP query, it is better to search the official IP segment on the Spider website to determine.

The access path can show the spider's access to your website. If you are an e-commerce website, you should pay attention to the crawling proportion of the next product page, so as to carry out targeted optimization.

The article crawling function can see whether it is included, the number of spider visits, and the number of inbound and outbound links.

Here we will focus on the pages that have spiders crawling but are not included, and consider whether to insert inbound and outbound links to build links.

Spider Analyzer plug-in download

Spider Analyzer is divided into free version and professional version. The following are the functional differences between the two versions.

The free version plug-in can be searched and installed directly from the WordPress background plug-in library.

Please download, upload and install the professional version after purchasing from the link below.

Download address