Load balancing with Apache: mod_proxy_balancer and health checks
mod_proxy_balancer turns Apache’s reverse proxy into a load balancer: one balancer:// name stands for a group of backend servers, and Apache shares requests between them. Add mod_proxy_hcheck and Apache also tests each backend in the background, takes a broken one out of rotation and puts it back once it recovers.
Modules you need
- mod_proxy and a protocol module such as mod_proxy_http.
- mod_proxy_balancer, which needs mod_slotmem_shm for shared memory.
- At least one scheduler: mod_lbmethod_byrequests (the default method), and optionally bybusyness, bytraffic or heartbeat.
- mod_proxy_hcheck for health checks (httpd 2.4.21 and later); it needs mod_watchdog.
- mod_status if you want the balancer-manager page.
# Debian/Ubuntu (slotmem_shm is pulled in automatically; watchdog is built in)
sudo a2enmod proxy proxy_http proxy_balancer lbmethod_byrequests lbmethod_bybusyness proxy_hcheck headers status
sudo systemctl restart apache2
# RHEL/Fedora: all of these are loaded by default
sudo apachectl -M | grep -E 'balancer|lbmethod|hcheck|slotmem'
A basic balancer
Three application servers on a private network serve the same app on port 8080:
<VirtualHost *:443>
ServerName www.example.com
# SSLEngine and certificate lines: see the HTTPS guide
<Proxy "balancer://app">
BalancerMember "http://10.0.0.11:8080"
BalancerMember "http://10.0.0.12:8080"
BalancerMember "http://10.0.0.13:8080"
ProxySet lbmethod=byrequests
</Proxy>
ProxyPreserveHost On
ProxyPass "/" "balancer://app/"
ProxyPassReverse "/" "balancer://app/"
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
Each BalancerMember is a backend (the documentation also calls them workers). Everything that works with a single ProxyPass target also works here, including timeouts and WebSockets (upgrade=websocket on the ProxyPass line). Test and reload with sudo apachectl configtest && sudo systemctl reload apache2 (httpd on RHEL).
Choosing a method
lbmethod= | Module | How it picks a backend | Good for |
|---|---|---|---|
byrequests (default) | mod_lbmethod_byrequests | Counts requests so that each member gets its weighted share | Similar requests, similar servers |
bybusyness | mod_lbmethod_bybusyness | Sends each new request to the member with the fewest requests in progress | Requests of very different length (reports, uploads, APIs) |
bytraffic | mod_lbmethod_bytraffic | Balances the bytes sent and received, not the number of requests | Downloads and large responses |
heartbeat | mod_lbmethod_heartbeat | Uses load reports that backends send with mod_heartbeat | Backends that are themselves Apache servers |
Weights. loadfactor (a number from 1 to 100, default 1) sets each member’s share. With loadfactor=1 on one server and loadfactor=3 on another, byrequests sends 1 of every 4 requests to the first and 3 to the second: out of 400 requests, 100 and 300. Only the ratio matters, so 25 and 75 would behave the same.
BalancerMember "http://10.0.0.11:8080" loadfactor=1
BalancerMember "http://10.0.0.12:8080" loadfactor=3
Sticky sessions
If your app keeps session data in the memory of one server, every request from a user must return to that server. mod_proxy_balancer does this with a cookie or a URL parameter that carries a route name. If the app does not set a suitable cookie, Apache can set one itself:
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<Proxy "balancer://app">
BalancerMember "http://10.0.0.11:8080" route=app1
BalancerMember "http://10.0.0.12:8080" route=app2
ProxySet stickysession=ROUTEID
</Proxy>
The cookie value .app1 starts with a dot because the balancer reads only the part after the dot as the route. The header line runs only when a route was missing or changed (BALANCER_ROUTE_CHANGED). Cookie names are case-sensitive.
Tomcat and other Java servers already add the server name to their session cookie (JSESSIONID=….node1). Set jvmRoute="node1" in Tomcat’s server.xml, use the same value as route, and cover both the cookie and the URL form:
ProxyPass "/" "balancer://tomcats/" stickysession=JSESSIONID|jsessionid scolonpathdelim=On
<Proxy "balancer://tomcats">
BalancerMember "http://10.0.0.21:8080" route=node1
BalancerMember "http://10.0.0.22:8080" route=node2
</Proxy>
If the backends cannot share sessions at all, add nofailover=On: then a user whose server fails gets an error instead of being sent to a server that does not know the session. Better still, store sessions in a database or cache shared by all backends; then you need no stickiness.
Failover, spares and standbys
- Error state and retry. When a member cannot be reached, Apache marks it as in error and tries another one. It retries the failed member after
retryseconds (default 60). failonstatus=502,503(set with ProxySet) also puts a member into error state when it returns those codes, for example while an app is restarting.status=+Hmakes a member a hot standby, used only when all others in its set are unavailable.status=+Rmakes a hot spare, which replaces one unusable member of the same set.lbset=1puts members in a second set that is used only when every member of set 0 is down, such as servers in another data center.status=+N(drain) takes only requests for existing sticky sessions, useful before maintenance.
<Proxy "balancer://app">
BalancerMember "http://10.0.0.11:8080"
BalancerMember "http://10.0.0.12:8080"
BalancerMember "http://10.0.0.19:8080" status=+H
BalancerMember "http://10.1.0.11:8080" lbset=1
ProxySet lbmethod=bybusyness failonstatus=502,503
</Proxy>
Health checks with mod_proxy_hcheck
Without health checks, Apache only notices a dead backend when a real user’s request fails. mod_proxy_hcheck checks members on its own schedule, independently of user traffic. A member that fails gets the flag C (HcFl) and receives no requests until it passes again.
ProxyHCExpr ok2xx {%{REQUEST_STATUS} =~ /^2/}
ProxyHCTemplate appcheck hcmethod=GET hcuri=/healthz hcexpr=ok2xx hcinterval=10 hcfails=2 hcpasses=2
<Proxy "balancer://app">
BalancerMember "http://10.0.0.11:8080" hctemplate=appcheck
BalancerMember "http://10.0.0.12:8080" hctemplate=appcheck
BalancerMember "http://10.0.0.13:8080" hctemplate=appcheck
</Proxy>
hcmethod:TCPonly checks that the port accepts connections;HEAD,GETandOPTIONSsend an HTTP/1.0 request;GET11,HEAD11andOPTIONS11(2.4.55 and later) use HTTP/1.1.hcuri=/healthzis added to the member URL. Point it at a cheap page that also checks the app’s database connection, not at the home page.hcexprnames aProxyHCExprtest. Without one, any 2xx or 3xx status passes.hc('body')lets you test the response body, for example{hc('body') !~ /maintenance/}.- The timing: with
hcinterval=10andhcfails=2, a dead server leaves rotation after about 20 seconds; withhcpasses=2it returns after two good checks, again about 20 seconds. ProxyHCTemplatekeeps the settings in one place; you can also write the parameters directly on each BalancerMember.
The balancer-manager
The balancer-manager page shows every member with its status, load and errors, and lets you change weights, drain or disable members and even add new ones while the server runs. Because it can change live routing, it must never be public:
<Location "/balancer-manager">
SetHandler balancer-manager
Require ip 10.0.0.0/24
</Location>
ProxyPass "/balancer-manager" "!"
Put the ProxyPass … "!" exclusion before your ProxyPass "/" … line; otherwise the request is sent to the backends instead of the manager. For access from outside the admin network, add a password as well (see password protection) and use HTTPS only. The page uses a random nonce against cross-site request forgery; do not turn it off with nonce=None. Changes made in the manager are lost on restart unless you set BalancerPersist On, and only balancers defined outside <Location> blocks can be managed. For read-only monitoring, mod_status is enough (see server status).
Check that it works
Add a temporary header that names the member used for each response:
Header set X-Backend "%{BALANCER_WORKER_NAME}e"
for i in 1 2 3 4 5 6; do
curl -s -o /dev/null -D - https://www.example.com/ | grep -i x-backend
done
Without stickiness you should see the members take turns, for example x-backend: http://10.0.0.11:8080, then …12…, then …13…. Stop one backend, wait for the health check interval, and confirm it disappears from the list and shows HcFl in the balancer-manager. Remove the header when you are done, since it reveals internal addresses.
Common mistakes
- “Failed to lookup provider 'shm' for 'slotmem': is mod_slotmem_shm loaded??” at startup: exactly what it says; load mod_slotmem_shm.
- Missing lbmethod module.
lbmethod=bybusynessneeds mod_lbmethod_bybusyness loaded; otherwise the configuration test fails with “unknown lbmethod”. - Sticky sessions that do not stick. The cookie name in
stickysessionis case-sensitive, Apache uses the part of the cookie value after the dot as the route, and every member needs a matchingroute=. Log%{BALANCER_SESSION_STICKY}eand%{BALANCER_WORKER_ROUTE}eto see what Apache found. - One Apache in front of many backends. The balancer is now the single point of failure. For high availability, run two Apache servers with a floating IP address (for example with keepalived) or a load balancer in front.
- Backends reachable directly. Firewall the backend ports so that only the balancer can reach them.
Sources
- mod_proxy_balancer, Apache httpd 2.4 documentation
- mod_proxy_hcheck, Apache httpd 2.4 documentation
- mod_proxy (BalancerMember and balancer parameters), Apache httpd 2.4 documentation
- Reverse Proxy Guide (balancers, failover, status flags), Apache httpd 2.4 documentation
- mod_lbmethod_byrequests, Apache httpd 2.4 documentation
- mod_lbmethod_bybusyness, Apache httpd 2.4 documentation
- mod_lbmethod_bytraffic, Apache httpd 2.4 documentation
- mod_lbmethod_heartbeat, Apache httpd 2.4 documentation
- mod_headers, Apache httpd 2.4 documentation
- proxy_balancer.load in the Debian apache2 package, Debian Sources
- debian/rules for apache2 (static modules), Debian Sources
- 00-proxy.conf in the Fedora httpd package, Fedora Package Sources
- mod_proxy_balancer.c source (2.4.x branch), apache/httpd on GitHub
- mod_proxy.c source (2.4.x branch), apache/httpd on GitHub