Classified directory archiving: Keepalived

Keepalive Nginx dual network (internal and external network) fault asynchronous drift dual active dual main mode (actual combat)

1 reply

Introduction:

With a high-performance combination like keepalived+Lvs, why do we need keepalived+Nginx. Keepalived is designed for Lvs. Lvs is a four tiered load balancing device. Although it has the advantage of high performance, it does not have a back-end server health check mechanism. Keepalived provides a series of health check mechanisms for lvs, such as TCP_CHECK, UDP_CHECK, HTTP_GET, etc. At the same time, lvs can also write its own health check script. Or combine ldirectory to implement backend health detection. However, LVS has always been unable to get rid of the fact that it is a four layer device and cannot parse the upper layer protocol. Nginx is different. Nginx is a 7-tier device that can parse the 7-tier protocol, filter some requests, and cache the request results. These are the unique advantages of Nginx. However, keepalived does not provide health detection for Nginx. You need to write some footsteps to conduct health inspection.

The following mainly explains the Keepalived+Nginx mode, excluding lvs. If it is not a large load, LVS is generally unavailable. Of course, you can also refer to:《 Keepalive LVS-DR Nginx single network dual active dual main configuration mode (actual combat) 》Article.

Prepare four servers or virtual machines:

Web Nginx Intranet: 10.16.8.8/10.16.8.9

Keepalived Intranet: 10.16.8.10 (ka67)/10.16.8.11 (ka68)
Keepalived public network: 172.16.8.10/172.16.8.11

Keepalived Intranet VIP: 10.16.8.100/10.16.8.101
Keepalived public network VIP: 172.16.8.100/172.16.8.101

OS：CentOS Linux release 7.4.1708 (Core)

precondition:

Install keepalived.
Time synchronization.
Set SELinux and firewall.
Between each other /etc/hosts Add the opposite host name to the file (optional).
Confirm that the network interface supports multicast (multicast) by default.

For the above deployment, please refer to:《 Keepalived installation and configuration file explanation 》。

1. ka67 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 vrrp_mcast_group4 224.0.0.111 } vrrp_instance External_1 { state MASTER interface eth1 virtual_router_id 171 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"   } vrrp_instance External_2 { state BACKUP interface eth1 virtual_router_id 172 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"   } vrrp_instance Internal_1 { state MASTER interface eth0 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           } vrrp_instance Internal_2 { state BACKUP interface eth0 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           }

2. ka68 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 vrrp_mcast_group4 224.0.0.111 } vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           } vrrp_instance External_2 { state MASTER interface eth1 virtual_router_id 172 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           } vrrp_instance Internal_1 { state BACKUP interface eth0 virtual_router_id 191 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           } vrrp_instance Internal_2 { state MASTER interface eth0 virtual_router_id 192 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           }

3. Create a common script for detection

 $ vim /usr/local/keepalived/etc/keepalived/notify.sh
 #!/ bin/bash # contact=' root@localhost ' notify() { local mailsubject="$(hostname) to be $1, vip floating" local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master    ;; backup) notify backup systemctl start nginx #After configuring here, the Nginx service can automatically start if it hangs ;; fault) notify fault     ;; *) echo "Usage: $(basename $0) {master|backup|fault}" exit 1 ;; esac

4. Start the keepalived service and test

Check the network card status after starting ka67:

 [ root@ka67  ~]# systemctl start keepalived
 [ root@ka67  ~]# ip a
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.100/32 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.101/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::436e:b837:43b:797c/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.100/32 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.101/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::1261:7633:b595:7719/64 scope link valid_lft forever preferred_lft forever

When ka68 is not started, ka67 adds four VIPs:

Public network eth0:

172.16.8.100/32
172.16.8.101/32

Intranet eth1:

10.16.8.100/32
10.16.8.101/32

Check the network card status after starting ka68:

 [ root@ka68  ~]# systemctl start keepalived
 [ root@ka68  ~]# ip a
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:79 brd ff:ff:ff:ff:ff:ff inet 172.16.8.11/24 brd 103.28.204.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.101/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::3d2c:ecdc:5e6d:70ba/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:82 brd ff:ff:ff:ff:ff:ff inet 10.16.8.11/24 brd 10.16.8.255 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.101/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::4fb3:d0a8:f08c:4536/64 scope link valid_lft forever preferred_lft forever

Ka68 added two VIPs, namely:

Public network eth0:

172.16.8.101/32

Intranet eth1:

10.16.8.101/32

Check the network card status information of ka67 again:

 [ root@ka67  ~]# ip a
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.100/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::436e:b837:43b:797c/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.100/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::1261:7633:b595:7719/64 scope link valid_lft forever preferred_lft forever

be aware 172.16.8.101/10.16.8.101 It has been removed. At this time, no matter whether any server is stopped, the 4 VIPs will not stop communication.

In addition, you can ka67/ka68 View the heartbeat status of the multicast address through the following command:

 [ root@ka67  ~]# tcpdump -nn -i eth1 host 224.0.0.111
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 02:00:15.690389 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20 02:00:15.692654 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20 02:00:16.691552 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20 02:00:16.693814 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20 02:00:17.692710 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20

up to now, vrrp High availability configuration of&test completed, let's continue to configure Web Nginx service.

5. Install and configure Nginx

In the back-end server 10.16.8.8/10.16.8.9 Install Nginx:

For Nginx see:《 Centos 7 source code compilation and installation Nginx 》。

Or by yum Install Nginx; Simple and fast:

 $ yum install epel-release -y $ yum install nginx -y

In order to distinguish different machines in the test environment, the server IP address is set for the web page, but the content obtained in the production environment is consistent.

Respectively in 10.16.8.8/10.16.8.9 Execute the following command:

 $ echo "Server 10.16.8.8" > /usr/share/nginx/html/index.html $ echo "Server 10.16.8.9" > /usr/share/nginx/html/index.html

Test whether the access is normal:

 $ curl //10.16.8.8 Server 10.16.8.8

Respectively in ka67/ka68 Install Nginx on the. I use it here yum Installation:

 $ yum install nginx psmisc -y

explain: psmisc Includes: fuser , killall , pstree And so on.

stay ka67/ka68 Configure Nginx on:

Backup default configuration file:

 $ mv /etc/nginx/conf.d/default.conf{,.bak} $ mv /etc/nginx/nginx.conf{,.bak}

Respectively in ka67/ka68 Add the following content to the nginx main configuration file:

 $ vim /etc/nginx/nginx.conf
 user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; } http { log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log  /var/log/nginx/access.log  main; sendfile            on; tcp_nopush          on; tcp_nodelay         on; keepalive_timeout   65; types_hash_max_size 2048; include             /etc/nginx/mime.types; default_type        application/octet-stream; include /etc/nginx/conf.d/*.conf; upstream webserverapps { server 10.16.8.8:80; server 10.16.8.9:80; #server 127.0.0.1:8080 backup; }

 server { listen 80; server_name _; location / { proxy_pass //webserverapps; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; add_header Access-Control-Allow-Origin *; } } }

Note: The blue part is mainly added to the above configuration. Other defaults are only for testing. Please adjust the configuration of the production environment according to your own needs.

stay ka67/ka68 Restart Nginx service:

 $ systemctl restart nginx

Test on ka67/ka68 respectively:

 [ root@ka67  ~]# for i in `seq 10`;  do curl 10.16.8.10;  done Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.9 Server 10.16.8.9

So far, Nginx reverse generation has also been realized. Next, we will combine Nginx with Keepalived to make Nginx support high availability.

6. Configure Keepalive Nginx high availability

Respectively in ka67/ka68 configuration file /usr/local/keepalived/etc/keepalived/keepalived.conf Global configuration block for global_defs Add below vrrp_script Configuration block:

 vrrp_script chk_nginx { script "killall -0 nginx" interval 2 weight -10 fall 2 rise 2 }

At all vrrp_instance Instance block, add track_script Blocks:

 track_script { chk_nginx }

For example:

 ... vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } track_script { chk_nginx } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_"/usr/local/keepalived/etc/keepalived/notify.sh fault" } ...

After configuration, restart ka67/ka68 's keepalived service:

 $ systemctl stop keepalived $ systemctl start keepalived

Summary:

During the configuration process, there is a problem that cannot be drifted and cross network segments. To solve the channel problem, we need to see more logs, analyze and judge more, and finally we can solve the problem. In any case, since you have chosen to keep alive, you should firmly believe in your original intention.
If you have any problem in the configuration process, please leave a message to solve the problem together.

Keepalive LVS-DR Nginx single network dual active dual main configuration mode (actual combat)

Comment

What is LVS/DR mode?

LVS is the abbreviation of Linux Virtual Server, which means Linux virtual server. It is a virtual server cluster system. LVS currently has three IP load balancing technologies (VS/NAT, VS/TUN, and VS/DR) and ten scheduling algorithms (rrr | wrr | lc | wlc | lblc | lblcr | dh | sh | sed | nq).

LVS exists in the Unix like system as a front end (Director), also known as a scheduler. It does not provide any services, but only accepts requests coming through the Internet and forwards them to the real server running in the background (RealServer) for processing, and then responds to the client.

LVS cluster adopts IP load balancing technology and content-based request distribution technology. The scheduler has a good throughput. It transfers requests evenly to different servers for execution, and the scheduler automatically masks server failures, thus forming a group of servers into a high-performance, highly available virtual server. The structure of the entire server cluster is transparent to the client, and there is no need to modify the client and server programs. Therefore, the transparency, scalability, high availability and manageability of the system should be considered in the design.

LVS has two important components: IPVS and IPVSADM. IPvs is the core component of LVS. It is just a framework, similar to iptables, and works in the kernel space. Ipvsadm is used to define the forwarding rules of LVS and works in the user space.

LVS has three forwarding types:

LVS-NAT mode:

It is called network address translation. It is relatively simple to implement. All RealServer cluster nodes and front-end scheduler directors must be in the same subnet. This model can implement port mapping. The operating system of RealServer can be any operating system. The front-end director needs to process both the requests from the client and the response information of the back-end RealServer, Forwarding the information responded by RealServer to the client, the front-end director can easily become the bottleneck of the performance of the entire cluster system. Generally, the IP address of the RealServer (hereinafter referred to as RIP) is a private address to facilitate communication between RealServer cluster nodes. Generally, the front-end director has two IP addresses, one is a VIP, which is a virtual IP address. The client sends a request to this IP address. One is DIP, which is the IP address of the real director. The RIP gateway should point to the director's DIP.

LVS-DR mode:

DR: Direct routing mode. This mode works through MAC address forwarding. All RealServer cluster nodes and front-end scheduler directors must be in the same physical network. This mode does not support port mapping. This mode has better performance than LVS-NAT. RIP can use the public network IP, and RIP gateway cannot point to DIP.

advantage:

Compared with the LVS/NAT mode, the DR mode does not need to forward the returned data through load balancing. If you want to take advantage of the DR mode, the number and length of the corresponding data packets should be far greater than the request data packets. Fortunately, most Web services have this feature, and the response and request are asymmetric. Therefore, common Web services can use this mode.

In this way, the load balancer is no longer the bottleneck of the system. If your load balancer only has a 100M full duplex network card and bandwidth, the horizontal expansion of the cluster can also enable the entire system to achieve 1G traffic.

The test results from the official LVS website also tell us that LVS-DR can accommodate more than 100 actual application servers, just for general services, which is enough.

Insufficient:

In DR mode, you cannot forward data across network segments. If you must load across network segments, you must use LVS/TUN mode.

LVS-TUN mode:

The RealServer server, known as the tunnel model, and the front-end director can be in different networks. This model also does not support port mapping. The RealServer can only use the operating systems that support IP tunnels. The front-end director only processes the client's requests, and then forwards the requests to the RealServer. The back-end RealServer directly responds to the client without passing through the director, RIP must not be a private IP. In DR and TUN modes, data packets are returned directly to users. Therefore, this address needs to be set on the Director Server and on each node of the cluster. This IP is usually bound to the loopback address on the Real Server, such as lo: 0. Similarly, on the Director Server, the virtual IP is bound to the real network interface device, such as eth0:0.

Start deployment:

Prepare four servers or virtual machines:

Web Nginx：10.16.8.8/10.16.8.9
Keepalived：10.16.8.10/10.16.8.11
Keepalived VIP：10.16.8.100/10.16.8.101
OS：CentOS Linux release 7.4.1708 (Core)

precondition:

Install keepalived.
Time synchronization.
Set SELinux and firewall.
Add each other's hostname to the/etc/hosts file (optional).
Confirm that the network interface supports multicast (multicast) by default.

For the above deployment, please refer to:《 Keepalived installation and configuration file explanation 》。

1. ka67 configuration file

 $ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 60 vrrp_mcast_group4 224.0.0.111 } vrrp_instance VI_1 { state MASTER interface eth1 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"         } vrrp_instance VI_2 { state BACKUP interface eth1 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          }

2. ka68 configuration file

 $ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 60 vrrp_mcast_group4 224.0.0.111 } vrrp_instance VI_1 { state BACKUP interface eth1 virtual_router_id 191 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          } vrrp_instance VI_2 { state MASTER interface eth1 virtual_router_id 192 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"         }

3. Create a general notify.sh detection script

Create this script separately:

 $ vim /usr/local/keepalived/etc/keepalived/notify.sh
 #!/ bin/bash # contact=' root@localhost ' notify() { local mailsubject="$(hostname) to be $1, vip floating" local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master    ;; backup) notify backup    ;; fault) notify fault     ;; *) echo "Usage: $(basename $0) {master|backup|fault}" exit 1 ;; esac

4. Start the keepalived service

 $ systemctl start keepalived $ systemctl enable keepalived

5. View multicast status

We can also view the multicast heartbeat status on any keepalived node through the tcpdump command, for example:

 $ tcpdump -nn -i eth1 host 224.0.0.111
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 00:32:31.714987 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20 00:32:31.715739 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20 00:32:32.716150 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20 00:32:32.716292 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20 00:32:33.717327 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20 00:32:33.721361 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20

If you are prompted with an error: -bash: tcpdump: command not found.

Install tcpdump:

 $ yum install tcpdump -y

6. Configure LVS

Install LVS separately. CentOS7 has integrated the core of LVS, so you only need to install LVS management tools:

 $ yum -y install ipvsadm

Stop the keepalived service of ka67/ka68 respectively:

 $ systemctl stop keepalived

Add the Virtual Server configuration at the end of the ka67/ka68 configuration file:

 $ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
 virtual_server 10.16.8.100 80 { delay_loop 3 lb_algo rr lb_kind DR protocol TCP # sorry_server 127.0.0.1 80 real_server 10.16.8.8 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } real_server 10.16.8.9 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } } virtual_server 10.16.8.101 80 { delay_loop 3 lb_algo rr lb_kind DR protocol TCP # sorry_server 127.0.0.1 80 real_server 10.16.8.8 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } real_server 10.16.8.9 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } }

7. Configure RS (Real Server) Web service

Install Apache Httpd or Nginx as web services on the web server, and install Nginx here.

For Nginx see:《 Centos 7 source code compilation and installation Nginx 》。

Or install Nginx in the following ways, which is simple and fast:

 $ yum install epel-release -y $ yum install nginx -y

In order to distinguish different machines in the test environment, the display page is set to the server IP address, but the content obtained in the production environment is consistent.

Execute the following commands on web8/web9 respectively:

 $ echo "Server 10.16.8.8" > /usr/share/nginx/html/index.html $ echo "Server 10.16.8.9" > /usr/share/nginx/html/index.html

Test whether the access is normal:

 $ curl //127.0.0.1 Server 10.16.8.8

8. Add RS script

Because some commands in this script are not available in Centos7 minimal installation, please install the network toolkit first:

 $ yum install net-tools -y

Add rs.sh scripts on the web server respectively:

 $ vim /tmp/rs.sh
 #!/ bin/bash vip1=10.16.8.100 vip2=10.16.8.101 dev1=lo:1 dev2=lo:2 case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $dev1 $vip1 netmask 255.255.255.255 broadcast $vip1 up ifconfig $dev2 $vip2 netmask 255.255.255.255 broadcast $vip2 up echo "VS Server is Ready!" ;; stop) ifconfig $dev down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce echo "VS Server is Cancel!" ;; *) echo "Usage `basename $0` start|stop" exit 1 ;; esac

Then start the script separately:

 $ /tmp/rs.sh start

If you need to stop, execute the following command:

 $ /tmp/rs.sh stop

9. Test

Test whether it can be accessed on another server

 [ root@localhost  ~]# for i in `seq 5`;  do >     curl 10.16.8.100 >     curl 10.16.8.101 > done Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.9 Server 10.16.8.8

According to the test results, the Keepalive+LVS-DR+Nginx high availability failover mode has been realized.

Keepalive dual network (internal and external network) fault synchronous drift dual active dual main mode

Comment

preface:

In the production environment, both the public network and the intranet are independent, so they are called dual networks. The public network and the private network realize synchronous drift in case of failure, such as: Keepalived+LVS-NAT Mode, you need to use vrrp_sync_group Set the synchronization drift group. In case of dual active and dual active, two VIPs need to be added at both ends to achieve the effect of mutual active and standby.

1. Schematic diagram:

  • The multicast IP address is 224.0.0.111.
  • The intranet VIP1 and the intranet VIP2 are the main and standby of each other.
  • Public network VIP1 and public network VIP2 are the main and standby of each other.
  • Intranet VIP1 and public VIP1 are a synchronization group.
  • Intranet VIP2 and public VIP2 are a synchronization group.
 +------+ |Client| +------+ /\ +--------+    |Internet| +--------+ /\ +--------+   |NAT network| +--------+ /\ +----------------------+ |Intranet VIP1: 10.16.8.100| |Intranet VIP2: 10.16.8.101| +----------------------+ /                \ +-----------------------+      +-----------------------+ |      KA+Lvs-NAT       |      |       KA+Lvs-NAT      | |Intranet VIP1: Master (eth1) | | Intranet VIP1: BACKUP (eth1)| |Intranet VIP2: BACKUP (eth1) | | Intranet VIP2: Master (eth1)| |Intranet: 10.16.8.10 (eth1) |<-->| Intranet: 10.16.8.11 (eth1)| |-----------------------|Multicast IP|-----------------------| |Public VIP1: Master (eth2) |<-->| Public VIP1: BACKUP (eth2)| |Public VIP2: BACKUP (eth2) | | Public VIP2: Master (eth2)| |Public network: 172.16.8.10 (eth2) | | Public network: 172.16.8.11 (eth2)| +-----------------------+      +-----------------------+ \                / +-----------------------+	  |Public network VIP1: 172.16.8.100| |Public network VIP2: 172.16.8.101| +-----------------------+ \/ +------+ |Resource pool| +------+

2. ka67 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka67 vrrp_mcast_group4 224.0.0.111 } vrrp_sync_group VG_1 { group { External_1 Internal_1 } } vrrp_sync_group VG_2 { group { External_2 Internal_2 } } vrrp_instance External_1 { state MASTER interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance External_2 { state BACKUP interface eth1 virtual_router_id 172 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state MASTER interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_2 { state BACKUP interface eth2 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

3. ka68 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka68 vrrp_mcast_group4 224.0.0.111 } vrrp_sync_group VG_1 { group { External_1 Internal_1 } } vrrp_sync_group VG_2 { group { External_2 Internal_2 } } vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance External_2 { state MASTER interface eth1 virtual_router_id 172 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state BACKUP interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_2 { state MASTER interface eth2 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

Keepalive dual network (internal and external network) fault asynchronous drift dual active dual main mode

Comment

preface:

In the production environment, both the public network and the intranet are independent, so they are called dual networks. The following configuration will avoid synchronous drift in case of intranet and public network failures, for example: Keepalived+LVS-DR 、 Keepalived+Nginx 、 Keepalived+HAProxy These do not need synchronous drift. in addition Keepalived+LVS-NAT Synchronous drift is required.

1. Schematic diagram:

  • The multicast IP address is 224.0.0.111.
  • The VIP internal and external networks of one machine are the main and standby.
 +------+ |Client| +------+ /\ +--------+    |Internet| +--------+ /\ +--------+   |NAT network| +--------+ /\ +----------------------+ |Intranet VIP1: 10.16.8.100| |Intranet VIP2: 10.16.8.101| +----------------------+ /                \ +-----------------------+      +-----------------------+ |KA+Lvs-DR/Nginx/HAProxy|      |KA+Lvs-DR/Nginx/HAProxy| |Intranet VIP1: Master (eth1) | | Intranet VIP1: BACKUP (eth1)| |Intranet VIP2: BACKUP (eth1) | | Intranet VIP2: Master (eth1)| |Intranet: 10.16.8.10 (eth1) |<-->| Intranet: 10.16.8.11 (eth1)| |-----------------------|Multicast IP|-----------------------| |Public VIP1: Master (eth2) |<-->| Public VIP1: BACKUP (eth2)| |Public VIP2: BACKUP (eth2) | | Public VIP2: Master (eth2)| |Public network: 172.16.8.10 (eth2) | | Public network: 172.16.8.11 (eth2)| +-----------------------+      +-----------------------+ \                / +-----------------------+	  |Public network VIP1: 172.16.8.100| |Public network VIP2: 172.16.8.101| +-----------------------+ \/ +------+ |Resource pool| +------+

2. ka67 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka67 vrrp_mcast_group4 224.0.0.111 } vrrp_instance External_1 { state MASTER interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance External_2 { state BACKUP interface eth1 virtual_router_id 172 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state MASTER interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_2 { state BACKUP interface eth2 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

3. ka68 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka68 vrrp_mcast_group4 224.0.0.111 } vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance External_2 { state MASTER interface eth1 virtual_router_id 172 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state BACKUP interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_2 { state MASTER interface eth2 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

Keepalive dual network (internal and external network) fault synchronous drift active standby single active mode

Comment

preface:

In the production environment, both the intranet and the public network are separate. To achieve synchronization drift between the intranet and the public network, such as the Keepalived+LVS-NAT mode, you need to set the vrrp_sync_group synchronization group. The difference is that the following configuration is only the active/standby mode, not the active/active mode.

1. Schematic diagram:

  • The multicast IP address is 224.0.0.111.
  • Master intranet and public network VIP belong to the same group.
  • The BACKUP intranet and public network VIP belong to the same group.
 +------+ |Client| +------+ /\ +--------+    |Internet| +--------+ /\ +--------+   |NAT network| +--------+ /\ +---------------------+ |Intranet VIP: 10.16.8.100| +---------------------+ /                \ +-----------------------+      +-----------------------+ |KA+Lvs/Nginx/HAProxy   |      |KA+Lvs/Nginx/HAProxy   | |Intranet VIP: Master (eth1) | | Intranet VIP: BACKUP (eth1)| |Intranet: 10.16.8.10 (eth1) |<-->| Intranet: 10.16.8.11 (eth1)| |-----------------------|Multicast IP|-----------------------| |Public network VIP: Master (eth2) |<-->| Public network VIP: BACKUP (eth2)| |Public network: 172.16.8.10 (eth2) | | Public network: 172.16.8.11 (eth2)| +-----------------------+      +-----------------------+ \                / +----------------------+	  |Public network VIP: 172.16.8.100| +----------------------+ \/ +------+ |Resource pool| +------+

2. ka67 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka67 vrrp_mcast_group4 224.0.0.111 } vrrp_sync_group VG_1 { group { External_1 Internal_1 } } vrrp_instance External_1 { state MASTER interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state MASTER interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

3. ka68 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka68 vrrp_mcast_group4 224.0.0.111 } vrrp_sync_group VG_1 { group { External_1 Internal_1 } } vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state BACKUP interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

Keepalive dual network (internal and external network) fault asynchronous drift active standby single active mode

Comment

preface:

In the production environment, the intranet and the public network are independent, so the intranet and the public network do not need to drift synchronously. For example, Keepalived+LVS-DR, Keepalived+Nginx, Keepalived+HAProxy do not need to drift synchronously.

Note: Keep alive+LVS-NAT mode is excluded.

1. Schematic diagram:

The multicast IP address is 224.0.0.111.

 +------+ |Client| +------+ /\ +--------+    |Internet| +--------+ /\ +--------+   |NAT network| +--------+ /\ +---------------------+ |Intranet VIP: 10.16.8.100| +---------------------+ /                \ +-----------------------+      +-----------------------+ |KA+Lvs/Nginx/HAProxy   |      |KA+Lvs/Nginx/HAProxy   | |Intranet VIP: Master (eth1) | | Intranet VIP: BACKUP (eth1)| |Intranet: 10.16.8.10 (eth1) |<-->| Intranet: 10.16.8.11 (eth1)| |-----------------------|Multicast IP|-----------------------| |Public network VIP: Master (eth2) |<-->| Public network VIP: BACKUP (eth2)| |Public network: 172.16.8.10 (eth2) | | Public network: 172.16.8.11 (eth2)| +-----------------------+      +-----------------------+ \                / +----------------------+	  |Public network VIP: 172.16.8.100| +----------------------+ \/ +------+ |Resource pool| +------+

2. ka67 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka67 vrrp_mcast_group4 224.0.0.111 } vrrp_sync_group VG_1 { group { External_1 Internal_1 } } vrrp_instance External_1 { state MASTER interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state MASTER interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

3. ka68 configuration file

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka68 vrrp_mcast_group4 224.0.0.111 } vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1     authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state BACKUP interface eth2 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

Keepalive single network dual active dual master configuration mode

Comment

preface:

Generally, this mode does not require a relatively complex configuration. Compared with the single network and single master mode, it has an additional active mode. It mainly realizes single network double main fault drift mode.

1. Architecture diagram:

The multicast IP address is 224.0.0.111.
NAT network can be configured according to the actual situation.

 +------+ |Client| +------+ /\ +--------+    |Internet| +--------+ /\ +--------+   |NAT network| +--------+ /\ +-----------------------+	  |Public network VIP1: 172.16.8.100| |Public network VIP2: 172.16.8.101| +-----------------------+ /                \ +-----------------------+      +-----------------------+ | KA+Lvs/Nginx/HAProxy  |      | KA+Lvs/Nginx/HAProxy  | |                       |<---->|                       | |VIP1: Master (eth1) | Multicast IP | VIP1: BACKUP (eth1)| | VIP2:BACKUP    (eth1) |<---->| VIP2:Master    (eth1) | | IP1:172.16.8.10(eth1) |      | IP1:172.16.8.11(eth1) | +-----------------------+      +-----------------------+ \                / +-----------------------+	  |Public network VIP1: 172.16.8.100| |Public network VIP2: 172.16.8.101| +-----------------------+ \/ +------+ |Resource pool| +------+

2. ka67 configuration file:

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka67 vrrp_mcast_group4 224.0.0.111 } vrrp_instance VG_1 { state MASTER interface eth0 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           } vrrp_instance VG_2 { state BACKUP interface eth0 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           }

3. ka68 configuration file:

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka68 vrrp_mcast_group4 224.0.0.111 } vrrp_instance VG_1 { state BACKUP interface eth0 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"   } vrrp_instance VG_2 { state MASTER interface eth0 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }

Keepalive single network active/standby/single active configuration mode (actual combat)

Comment

preface:

The following is described in Keepalived The simplest active/standby mode is configured in. Later, I will talk about the fault drift mode from the simple single network single active/standby mode to the dual network dual active/dual synchronous mode.

about Keepalived Introduction. It will not be described here. Please refer to the previous article:

《 Keep alive source code compilation, installation and configuration file explanation 》。

Architecture diagram:

The multicast IP address is 224.0.0.111.
NAT network can be configured according to the actual situation.

 +------+ |Client| +------+ /\ +--------+ |Internet| +--------+ /\ +--------+ |NAT network| +--------+ /\ +-----------------------+ |Public network VIP1: 172.16.8.100| +-----------------------+ /                \ +-----------------------+      +-----------------------+ | KA+Lvs/Nginx/HAProxy  |      | KA+Lvs/Nginx/HAProxy  | |VIP1: Master (eth1) | Multicast IP | VIP1: BACKUP (eth1)| | IP1:172.16.8.10(eth1) |      | IP1:172.16.8.11(eth1) | +-----------------------+      +-----------------------+ \                / +-----------------------+ |Public network VIP1: 172.16.8.100| +-----------------------+ \/ +------+ |Resource pool| +------+

Environmental Science:

MASTER：172.16.8.10
BACKUP：172.16.8.11
VIP：172.16.8.100
OS：CentOS Linux release 7.4.1708 (Core)

precondition:

  • Time synchronization.
  • set up SELinux And firewalls.
  • Between each other /etc/hosts Add the opposite host name to the file (optional).
  • Confirm that the interface supports multicast (multicast) by default.

《 Keep alive source code compilation, installation and configuration file explanation 》The above prerequisites have been completed.

1. Single network active/standby configuration file

MASTER configuration file:

 global_defs { notification_email { root@localhost } notification_email_from  ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka67 vrrp_mcast_group4 224.0.0.111 } vrrp_instance VG_1 { state MASTER interface eth0 virtual_router_id 103 priority one hundred advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"           }

BACKUP configuration file:

 global_defs { notification_email { root@localhost } notification_email_from  ka68@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka68 vrrp_mcast_group4 224.0.0.111 } vrrp_instance VG_1 { state BACKUP interface eth0 virtual_router_id 103 priority ninety-five advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"              }

2. General script

The following is notfiy.sh Common detection script:

 $ cat /usr/local/keepalived/etc/keepalived/notify.sh
 #!/ bin/bash contact=' root@localhost ' notify() { local mailsubject="$(hostname) to be $1, vip floating" local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master    ;; backup) notify backup    ;; fault) notify fault     ;; *) echo "Usage: $(basename $0) {master|backup|fault}" exit 1 ;; esac

3. Active/standby test

Test MASTER

Before starting keepalived, check the network card information:

 [ root@ka67  keepalived]# ip a
 ... eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::436e:b837:43b:797c/64 scope link valid_lft forever preferred_lft forever

After starting keepalived, check the network card information again:

 [ root@ka67  keepalived]# ip a
 ... eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.100 /32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::436e:b837:43b:797c/64 scope link valid_lft forever preferred_lft forever

VIP 172.16.8.100 has been successfully added.

Test MASTER

Start keepalived:

 [ root@ka68  keepalived]# systemctl start keepalived

Now stop MASTER and see if it will drift to BACKUP:

 [ root@ka67  keepalived]# systemctl stop keepalived

To view the BACKUP running log:

 [ root@ka68  keepalived]# cat /cat /var/log/messages
 ... Keepalived_vrrp[1451]: VRRP_Instance(VG_1) Transition to MASTER STATE Keepalived_vrrp[1451]: VRRP_Instance(VG_1) Entering MASTER STATE Keepalived_vrrp[1451]: VRRP_Instance(VG_1) setting protocol VIPs. Keepalived_vrrp[1451]: Sending gratuitous ARP on eth0 for 172.16.8.100 ...

Successfully drifted to BACKUP host.

Restart MASTER:

 [ root@ka67  keepalived]# systemctl start keepalived

To view the BACKUP Keepalived service status:

 [ root@ka68  keepalived]# systemctl status keepalived
 keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled) Active: active ( running ) since Tue 2018-03-02 22:13:14 EST;  15min ago Process: 1448 ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 1449 (keepalived) CGroup: /system.slice/keepalived.service ├─1449 /usr/local/keepalived/sbin/keepalived -D ├─1450 /usr/local/keepalived/sbin/keepalived -D └─1451 /usr/local/keepalived/sbin/keepalived -D Keepalived_vrrp[1451]: Sending gratuitous ARP on eth0 for 172.16.8.100 Keepalived_vrrp[1451]: VRRP_Instance(VG_1) Sending/queueing gratuitous ARPs on eth0 for 172.16.8.100 Keepalived_vrrp[1451]: Sending gratuitous ARP on eth0 for 172.16.8.100 Keepalived_vrrp[1451]: Sending gratuitous ARP on eth0 for 172.16.8.100 Keepalived_vrrp[1451]: Sending gratuitous ARP on eth0 for 172.16.8.100 Keepalived_vrrp[1451]: Sending gratuitous ARP on eth0 for 172.16.8.100 Keepalived_vrrp[1451]: VRRP_Instance(VG_1) Entering BACKUP STATE Keepalived_vrrp[1451]: VRRP_Instance(VG_1) removing protocol VIPs. Keepalived_vrrp[1451]: Opening script file /usr/local/keepalived/etc/keepalived/notify.sh

The above status indicates that when MASTER resumes service, BACKUP's Keepalived will automatically drift to MASTER. Because the weight value of MASTER is higher than BACKUP. The above is the drift from BACKUP to MASTER.

Keep alive source code compilation, installation and configuration file explanation

Comment

What is Keepalived?

Keepalived is a server high availability solution based on the vrrp protocol. It can be used to avoid a single point of IP failure. Similar tools include heartbeat and corosync. However, it does not appear alone, but works together with LVS, Nginx and HAproxy to achieve high availability.

What is the VRRP protocol?

Full name of VRRP Vritual Router Redundancy Protocol , virtual routing redundancy protocol. By combining several devices that provide routing functions into a virtual routing device, a certain mechanism is used to ensure the high availability of virtual routing, so as to maintain business continuity and reliability.

In a virtual router composed of configurations, there are MASTER and BACKUP. MASTER is the primary node. In a virtual router, there can only be one MASTER, but there can be multiple BACKUP nodes. BACKUP is the standby node. That is, after the master node fails, BACKUP takes over all resources of the MASTER node. When there are multiple BACKUP nodes priority (priority) to elect who is the substitute of MASTER. When the priority value of the BACKUP node is the same, it is determined according to the size of its IP address.

precondition:

  • The time between nodes must be synchronized.
  • ensure Firewalld and SELinux It will not become an obstacle.
  • The network interface used by each node for cluster service must support MULTICAST (Multicast) communication. Use Class D address (224-239). It is recommended to define the multicast address manually, because if multiple cluster services use the default, although there is an authentication mechanism, they will still send messages to each other, which may affect the performance and generate useless log information.

1. Time synchronization

See《 Centos 7 Chrony Set Server Cluster System Time Synchronization 》。

2. Firewall configuration

 $ firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface eth0 --destination 224.0.0.111 --protocol vrrp -j ACCEPT $ firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface eth0 --destination 224.0.0.111 --protocol vrrp -j ACCEPT $ firewall-cmd --reload

3. Enable route forwarding

 $ echo "net.ipv4.ip_forward=1" >>/etc/sysctl.conf $ echo 1 > /proc/sys/net/ipv4/ip_forward

4. Install keepalived

 $ cd /tmp $ wget //www.keepalived.org/software/keepalived-1.3.9.tar.gz $ tar xvf keepalived-1.3.9.tar.gz $ cd keepalived-1.3.9 $ ./ configure --prefix=/usr/local/keepalived $ make && make install $ ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin $ mkdir /etc/keepalived/ $ ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

The following errors may occur during installation:

Possible error 1:

 checking for gcc...  no checking for cc... no checking for cl.exe...  no configure: error: in `/tmp/keepalived-1.3.9': configure: error: no acceptable C compiler found in $PATH See `config.log' for more details

Solution:

 $ yum install gcc -y

Possible error 2:

 configure: error: !!!  OpenSSL is not properly installed on your system. !!! !!!  Can not include OpenSSL headers files.            !!!

Solution:

 $ yum install openssl-devel -y

Possible error 3:

 *** WARNING - this build will not support IPVS with IPv6.  Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

Solution:

 $ yum install libnl-devel

Possible error 4:

 checking libnfnetlink/libnfnetlink.h usability...  no checking libnfnetlink/libnfnetlink.h presence...  no checking for libnfnetlink/libnfnetlink.h... no configure: error: libnfnetlink headers missing

Solution:

 $ yum install libnfnetlink-devel -y

Installation summary:

For a Centos 7 system with minimal installation, when compiling and installing keepalived source code, first install the following dependency packages, and then compile and install keepalived, the above errors will not occur.

Install dependent packages:

 $ yum install gcc openssl-devel libnl-devel libnfnetlink-devel ipvsadm -y

5. Explanation of keepalived configuration file

 #Global configuration, including two sub configuration blocks: global definition and static address and route. !  Configuration File for keepalived Global_defs {# Global definition Notification_email {# Notification email related settings acassen@firewall.loc #Mail destination address } notification_email_from  rwl@renwole.com #From Smtp_server 127.0.0.1 # Use the local mail service Smtp_connect_timeout 30 # Set the timeout for connecting to the smtp server Router_id LVS_LEVEL # Identifies that the current node is unique and cannot be the same between nodes #It is time-consuming to check all addresses in the vrrp message #Setting this flag means that if the received message and the previous message come from the same router, the check will not be performed. Skip check by default vrrp_skip_check_adv_addr Vrrp_strict # Strictly implement the VRRP protocol specification. This mode does not support node unicast #Decimal type, in seconds. #The delay time between each group of Gratuitous arp messages on a network card. #The default value is 0. One sent message=n groups of arp messages vrrp_garp_interval 0 #Decimal type, in seconds #The delay time between each group of na messages on a network card is 0 by default vrrp_gna_interval 0 } #This area is VRRP configuration, including two sub configuration blocks: vrrp_sync_group/vrrp_instance, which is mainly used to provide external service VIP area and its related attributes Vrrp_instance VI_1 {# VRRP instance State MASTER # There can only be one MASTER, and the rest should be BACKUP; Interface eth1 # External network interface Virtual_router_id 65 # virtual route ID ID, number, must be the same as in backup Priority 100 # priority, the number must be greater than backup Advert_int 1 # Multicast message sending interval. The settings of the two nodes must be the same, in seconds Authentication {# Set the authentication information. The two nodes must be consistent (clear text) auth_type PASS auth_pass 1111 } #Virtual address, i.e. Floating IP Virtual_ipadress {# can be abbreviated to a single address, and the system will calculate the mask and device by default 172.16.28.65 # 172.16.28.65/24 # 172.16.28.65/24 dev eth1 } #Define notification script (add another) notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } #This area is LVS configuration. If Keepalived+LVS is used, this configuration is required. If others are used, such as Keepalived+Nginx, no configuration is required. #LVS contains two sub configuration blocks: virtual_server_group/virtual_server #Virtual_server: Virtual server. Each virtual server contains multiple real servers real_servers. Virtual_server 172.16.28.65 80 {# Virtual IP listens to port 80 Delay_loop 6 # Health check interval, seconds Lb_algo rr # Load scheduling algorithm, usually using wlc or rr Lb_kind NAT # LVS load forwarding rules, DR, NAT, TUN, etc Persistence_timeout 50 # Session duration, seconds Protocol TCP # There are two forwarding protocols: tcp and udp Real_server 172.16.28.65 80 {# Configure the address and port of the real server Weight 1 # Weight SSL_GET { url { Path/# Health check page Digest ff20ad2481f97b1754ef3e12ecd3a9cc # MD5 value calculated } url { path /mrtg/ digest 9b3a0c85a887a256d6939da88aabd8cd } Connect_timeout 3 # Connection timeout, seconds Nb_get_retry 3 # Number of failed retries, remove after exceeding Delay_before_retry 3 # Failure retry interval, seconds } } } ...

6. Start keepalived

After the keepalived.conf is correctly configured, you can start keepalived and join the boot auto start service.

 $ systemctl start keepalived $ systemctl enable keepalived

The compilation and installation of keepalived source code is completed.

Note: This article mainly introduces what is keepalived and how to install the keepalived and annotation keepalived configuration files. There is no specific experimental or production configuration.

To learn about the various modes of keepalived in the production environment, please refer to the following articles:

《 Keepalive single network active/standby/single active configuration mode (actual combat) 》
《 Keepalive single network dual active dual master configuration mode 》
《 Keepalive dual network (internal and external network) fault asynchronous drift active standby single active mode 》
《 Keepalive dual network (internal and external network) fault synchronous drift active standby single active mode 》
《 Keepalive dual network (internal and external network) fault asynchronous drift dual active dual main mode 》
《 Keepalive dual network (internal and external network) fault synchronous drift dual active dual main mode 》