Virtual down, server up
From LB Wiki
Problem
The load balancer's health checking shows there are available servers for the virtual service, however, you try to browse or otherwise connect to the virtual service, and nothing happens. The servers are up, but it looks like the virtual service is down. What gives?
This is perhaps the most common situation I run into with load balancing, and it has to do with how traffic flows (check out the 4-step NAT page).
The issue is typically one of traffic flow. For load balancing to work traffic needs to go through the load balancer on the way in, and on the way out DSR is the exception).
Traffic is forced through the load balancer on the way in because the load balancer has the VIP (virtual IP), so traffic naturally flows there. To force the load balancer to be in the path of traffic on the way out, one of three things is done typically:
- Route-path (Transparent): The load balancer is the default gateway of the servers
- Route-path (Non-Transparent): Inbound traffic is made to look like it's coming from the load balancer, so the servers respond to the load balancer
- Bridge-path: The load balancers acts as a Layer 2 bridge between the server and client networks.
Virtually all of the load balancers are capable of the first method, most support second as well. Many of the switch-based load balancers (such as A10 Networks or Nortel's Alteon) do all three.
Fix
The fix is generally to change the default route on the real server to that of the load balancer. Alternatively, you can set the Virtual Service to a non-transparency.
