Introduction:
With a high-performance combination like keepalived+Lvs, why do we need keepalived+Nginx. Keepalived is designed for Lvs. Lvs is a four tiered load balancing device. Although it has the advantage of high performance, it does not have a back-end server health check mechanism. Keepalived provides a series of health check mechanisms for lvs, such as TCP_CHECK, UDP_CHECK, HTTP_GET, etc. At the same time, lvs can also write its own health check script. Or combine ldirectory to implement backend health detection. However, LVS has always been unable to get rid of the fact that it is a four layer device and cannot parse the upper layer protocol. Nginx is different. Nginx is a 7-tier device that can parse the 7-tier protocol, filter some requests, and cache the request results. These are the unique advantages of Nginx. However, keepalived does not provide health detection for Nginx. You need to write some footsteps to conduct health inspection.
The following mainly explains the Keepalived+Nginx mode, excluding lvs. If it is not a large load, LVS is generally unavailable. Of course, you can also refer to:《 Keepalive LVS-DR Nginx single network dual active dual main configuration mode (actual combat) 》Article.
Prepare four servers or virtual machines:
Web Nginx Intranet: 10.16.8.8/10.16.8.9
Keepalived Intranet: 10.16.8.10 (ka67)/10.16.8.11 (ka68)
Keepalived public network: 172.16.8.10/172.16.8.11
Keepalived Intranet VIP: 10.16.8.100/10.16.8.101
Keepalived public network VIP: 172.16.8.100/172.16.8.101
OS:CentOS Linux release 7.4.1708 (Core)
precondition:
Install keepalived.
Time synchronization.
Set SELinux and firewall.
Between each other /etc/hosts
Add the opposite host name to the file (optional).
Confirm that the network interface supports multicast (multicast) by default.
For the above deployment, please refer to:《 Keepalived installation and configuration file explanation 》。
1. ka67 configuration file
global_defs { notification_email { root@localhost } notification_email_from ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 vrrp_mcast_group4 224.0.0.111 } vrrp_instance External_1 { state MASTER interface eth1 virtual_router_id 171 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance External_2 { state BACKUP interface eth1 virtual_router_id 172 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state MASTER interface eth0 virtual_router_id 191 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_2 { state BACKUP interface eth0 virtual_router_id 192 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }
2. ka68 configuration file
global_defs { notification_email { root@localhost } notification_email_from ka@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 vrrp_mcast_group4 224.0.0.111 } vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance External_2 { state MASTER interface eth1 virtual_router_id 172 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole1 } virtual_ipaddress { 10.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_1 { state BACKUP interface eth0 virtual_router_id 191 priority 95 advert_int 1 authentication { auth_type PASS auth_pass renwole2 } virtual_ipaddress { 172.16.8.100 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" } vrrp_instance Internal_2 { state MASTER interface eth0 virtual_router_id 192 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole3 } virtual_ipaddress { 172.16.8.101 } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault" }
3. Create a common script for detection
$ vim /usr/local/keepalived/etc/keepalived/notify.sh
#!/bin/bash # contact=' root@localhost ' notify() { local mailsubject="$(hostname) to be $1, vip floating" local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup systemctl start nginx #After configuring here, the Nginx service can automatically start if it hangs ;; fault) notify fault ;; *) echo "Usage: $(basename $0) {master|backup|fault}" exit 1 ;; esac
4. Start the keepalived service and test
Check the network card status after starting ka67:
[ root@ka67 ~]# systemctl start keepalived
[ root@ka67 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.100/32 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.101/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::436e:b837:43b:797c/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.100/32 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.101/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::1261:7633:b595:7719/64 scope link valid_lft forever preferred_lft forever
When ka68 is not started, ka67 adds four VIPs:
Public network eth0:
172.16.8.100/32
172.16.8.101/32
Intranet eth1:
10.16.8.100/32
10.16.8.101/32
Check the network card status after starting ka68:
[ root@ka68 ~]# systemctl start keepalived
[ root@ka68 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:79 brd ff:ff:ff:ff:ff:ff inet 172.16.8.11/24 brd 103.28.204.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.101/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::3d2c:ecdc:5e6d:70ba/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:82 brd ff:ff:ff:ff:ff:ff inet 10.16.8.11/24 brd 10.16.8.255 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.101/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::4fb3:d0a8:f08c:4536/64 scope link valid_lft forever preferred_lft forever
Ka68 added two VIPs, namely:
Public network eth0:
172.16.8.101/32
Intranet eth1:
10.16.8.101/32
Check the network card status information of ka67 again:
[ root@ka67 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0 valid_lft forever preferred_lft forever inet 172.16.8.100/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::436e:b837:43b:797c/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1 valid_lft forever preferred_lft forever inet 10.16.8.100/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::1261:7633:b595:7719/64 scope link valid_lft forever preferred_lft forever
be aware 172.16.8.101/10.16.8.101
It has been removed. At this time, no matter whether any server is stopped, the 4 VIPs will not stop communication.
In addition, you can ka67/ka68
View the heartbeat status of the multicast address through the following command:
[ root@ka67 ~]# tcpdump -nn -i eth1 host 224.0.0.111
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 02:00:15.690389 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20 02:00:15.692654 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20 02:00:16.691552 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20 02:00:16.693814 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20 02:00:17.692710 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20
up to now, vrrp
High availability configuration of&test completed, let's continue to configure Web
Nginx service.
5. Install and configure Nginx
In the back-end server 10.16.8.8/10.16.8.9
Install Nginx:
For Nginx see:《 Centos 7 source code compilation and installation Nginx 》。
Or by yum
Install Nginx; Simple and fast:
$ yum install epel-release -y $ yum install nginx -y
In order to distinguish different machines in the test environment, the server IP address is set for the web page, but the content obtained in the production environment is consistent.
Respectively in 10.16.8.8/10.16.8.9
Execute the following command:
$ echo "Server 10.16.8.8" > /usr/share/nginx/html/index.html $ echo "Server 10.16.8.9" > /usr/share/nginx/html/index.html
Test whether the access is normal:
$ curl //10.16.8.8 Server 10.16.8.8
Respectively in ka67/ka68
Install Nginx on the. I use it here yum
Installation:
$ yum install nginx psmisc -y
explain: psmisc
Includes: fuser
, killall
, pstree
And so on.
stay ka67/ka68
Configure Nginx on:
Backup default configuration file:
$ mv /etc/nginx/conf.d/default.conf{,.bak} $ mv /etc/nginx/nginx.conf{,.bak}
Respectively in ka67/ka68
Add the following content to the nginx main configuration file:
$ vim /etc/nginx/nginx.conf
user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; } http { log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; upstream webserverapps { server 10.16.8.8:80; server 10.16.8.9:80; #server 127.0.0.1:8080 backup; }
server { listen 80; server_name _; location / { proxy_pass //webserverapps; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; add_header Access-Control-Allow-Origin *; } } }
Note: The blue part is mainly added to the above configuration. Other defaults are only for testing. Please adjust the configuration of the production environment according to your own needs.
stay ka67/ka68
Restart Nginx service:
$ systemctl restart nginx
Test on ka67/ka68 respectively:
[ root@ka67 ~]# for i in `seq 10`; do curl 10.16.8.10; done Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.8 Server 10.16.8.9 Server 10.16.8.9 Server 10.16.8.9
So far, Nginx reverse generation has also been realized. Next, we will combine Nginx with Keepalived to make Nginx support high availability.
6. Configure Keepalive Nginx high availability
Respectively in ka67/ka68
configuration file /usr/local/keepalived/etc/keepalived/keepalived.conf
Global configuration block for global_defs
Add below vrrp_script
Configuration block:
vrrp_script chk_nginx { script "killall -0 nginx" interval 2 weight -10 fall 2 rise 2 }
At all vrrp_instance
Instance block, add track_script
Blocks:
track_script { chk_nginx }
For example:
... vrrp_instance External_1 { state BACKUP interface eth1 virtual_router_id 171 priority 100 advert_int 1 authentication { auth_type PASS auth_pass renwole0 } virtual_ipaddress { 10.16.8.100 } track_script { chk_nginx } notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup" notify_"/usr/local/keepalived/etc/keepalived/notify.sh fault" } ...
After configuration, restart ka67/ka68
's keepalived service:
$ systemctl stop keepalived $ systemctl start keepalived
Summary:
During the configuration process, there is a problem that cannot be drifted and cross network segments. To solve the channel problem, we need to see more logs, analyze and judge more, and finally we can solve the problem. In any case, since you have chosen to keep alive, you should firmly believe in your original intention.
If you have any problem in the configuration process, please leave a message to solve the problem together.