====== [HOWTO] Balancing Gateways (keepalived+haproxy) ====== ^ Documentation ^| ^Name:| [HOWTO] Balancing Gateways (keepalived+haproxy) | ^Description:| A production-ready way to balance gateways | ^Modification date :|28/07/2019| ^Owner:|dodger| ^Notify changes to:|Owner | ^Tags:|ceph, object storage | ^Scalate to:|The_fucking_bofh| ====== Pre-Requirements ====== * [[linux:ceph:howtos:using_amazon_dns_bucket_naming|Setup S3 naming method]] ====== Instructions ====== ===== Deploy lb nodes ===== For example: bash CloneWars.sh -c nuciberterminal -h AVMLP-OSLB-001 -i 10.20.54.1 -m 20 -O -r 8192 -v 2 -o 4 -F bash CloneWars.sh -c nuciberterminal2 -h AVMLP-OSLB-002 -i 10.20.54.2 -m 20 -O -r 8192 -v 2 -o 4 -F ===== Apply basic states ===== export THESERVER="avmlp-oslb-0*" salt "${THESERVER}" state.apply salt "${THESERVER}" state.apply nsupdate ===== Install required packages ===== salt "${THESERVER}" pkg.install haproxy salt "${THESERVER}" pkg.install keepalived ===== Setup HAproxy ===== Sample from clover: #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 #log /var/log/haproxy.log chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 2048 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 2048 frontend http_web *:80 mode http default_backend rgw #frontend rgw­-https # bind :443 ssl crt /etc/ssl/private/example.com.pem # default_backend rgw backend rgw balance roundrobin mode http server rgw1 avmlp-osgw-001.ciberterminal.net:80 check maxconn 512 server rgw2 avmlp-osgw-002.ciberterminal.net:80 check maxconn 512 server rgw3 avmlp-osgw-003.ciberterminal.net:80 check maxconn 512 server rgw4 avmlp-osgw-004.ciberterminal.net:80 check maxconn 512 ===== Setup rsyslog ===== Necessary for haproxy logging: --- rsyslog.conf 2018-10-30 15:49:15.000000000 +0100 +++ rsyslog.conf.new 2019-08-13 17:43:26.004833747 +0200 @@ -12,12 +12,12 @@ #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception -#$ModLoad imudp -#$UDPServerRun 514 +$ModLoad imudp +$UDPServerRun 514 # Provides TCP syslog reception -#$ModLoad imtcp -#$InputTCPServerRun 514 +$ModLoad imtcp +$InputTCPServerRun 514 #### GLOBAL DIRECTIVES #### @@ -72,6 +72,8 @@ # Save boot messages also to boot.log local7.* /var/log/boot.log +# haproxy logging +local2.* /var/log/haproxy.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding ===== Setup keepalived ===== global_defs { notification_email { dodger@ciberterminal.net } notification_email_from clover@ciberterminal.net smtp_server mta4.bavel.biz smtp_connect_timeout 30 ! router_id LVS_DEVEL ! vrrp_skip_check_adv_addr ! vrrp_strict ! vrrp_garp_interval 0 ! vrrp_gna_interval 0 } vrrp_script chk_haproxy { script "killall -0 haproxy" # check the haproxy process interval 2 # every 2 seconds weight 2 # add 2 points if OK } vrrp_instance VI_1 { interface eth0 # interface to monitor state MASTER # MASTER on haproxy, BACKUP on haproxy2 virtual_router_id 51 priority 101 # 101 on haproxy, 100 on haproxy2 virtual_ipaddress { 10.20.54.0 # virtual ip address } track_script { chk_haproxy } smtp_alert } On the secondary node, you'll have to chante the line: state MASTER # MASTER on haproxy, BACKUP on haproxy2 ===== setup pmta to allow sending un-authenticated emails ===== # avmlp-oslb-001 always-allow-relaying yes default-virtual-mta operativa smtp-service yes require-auth false dsn-return-default full # avmlp-oslb-002 always-allow-relaying yes default-virtual-mta operativa smtp-service yes require-auth false dsn-return-default full # clover.ciberterminal.net always-allow-relaying yes default-virtual-mta operativa smtp-service yes require-auth false dsn-return-default full ===== Restart & enable all ===== systemctl restart rsyslog systemctl restart haproxy systemctl restart keepalived.service systemctl enable rsyslog systemctl enable haproxy systemctl enable keepalived.service ====== Official documentation ====== * [[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/ceph_example|Load Balancing Ceph Object Gateway Servers with HAProxy and Keepalived]]