Compare balancing policies

How should the balancer pick an upstream? There’s no universally right answer, only policies that fit or fight your workload.

The three families you’ll actually use: round robin rotates requests through upstreams in order. Least connections favors the currently less busy upstream. Hash-based policies can keep related clients or requests sticky to the same upstream, by hashing the client IP, a header, or a cookie.

Make selection deterministic

Start with round robin, because its behavior is predictable enough to verify by eye. Make selection visibly deterministic:

reverse_proxy 127.0.0.1:4001 127.0.0.1:4002 {
  lb_policy round_robin
}

Send an even request count and count the ports:

for i in {1..10}; do
  curl --silent http://127.0.0.1:8080/ | grep -o '"port":[0-9]*'
done | sort | uniq -c
#   5 "port":4001
#   5 "port":4002

Exactly even, unlike the random default from the previous lesson. Round robin’s weakness is that it’s blind: a request going to a struggling upstream counts the same as one going to an idle upstream.

Expose round robin’s blind spot

Create a slow endpoint and compare least_conn. Add a route to the lab backend that sleeps for five seconds, then hold a few slow requests open against round robin — quick requests keep landing on the busy upstream anyway, right behind the stuck ones.

Switch the policy:

lb_policy least_conn

Repeat the experiment. While one upstream holds the slow connections, new requests drain to the other one. least_conn reacts to actual load, which makes it a better default when request costs vary a lot.

Whichever policy you test, record distribution instead of assuming it. The counting loop is your instrument.

Sticky selection, and its cost

Sometimes requests aren’t interchangeable because the server holds per-client session state in memory. Then you need stickiness: lb_policy client_ip_hash keeps a client on one upstream, and lb_policy cookie pins browsers via a cookie.

Use it knowingly, not by default. Sticky routing can hide server-side session design problems and produce uneven load — one heavy client saturates one upstream while others idle, and when that upstream dies, its sessions die with it. Prefer shared or externalized state when practical (a database, Redis), so any upstream can serve any client and the balancer stays free to balance.