Your load balancer usually offers this as configuration. Using HAProxy as an example (this is similar for most load balancers) you can change:
- Whether a server is in balance or not
- What proportion of traffic it should receive
- What strategy to use to divide traffic between servers
Common use cases for this would be:
- Taking a server out of balance (and allowing existing connections to drain) before running a deployment, then warming it back up with a slower rate of traffic.
- Splitting a small proportion of traffic for a canary release, which you might also use "session pinning" for, to make sure a canary user stays on the canary server, rather than flip-flopping between the old and new version
With HAProxy, you can perform some operations via the command line to take servers in and out of balance. For the load balancing strategy, you'd normally change the config and reload it (which, again, you could automate via the command line).
It's not uncommon for people to set up HAProxy with "redispatch" configured. They can then surprise it by deploying to a server without telling it. HAProxy will receive the failure and re-send the request to a different server. The user has no idea there was any issue. You need to make sure this doesn't cause a domino effect in your scenario.
Nginx is pretty similar. Other load balancers offer things like REST APIs for these operations.