Circuit Breaker

One of the properties of a remote call is that it can fail, or hang without a response until some timeout limit is reached. If there are many callers of this unresponsive service this can result in the caller run out of critical resources leading to cascading failures across multiple systems. To prevent this kind of catastrophic cascade we can use a circuit breaker.

The basic idea behind the circuit breaker is that you wrap external calls in a circuit breaker object. The circuit breaker object monitors for failures and once the failures reach a certain threshold, the circuit breaker trips, and all further calls via the circuit breaker return an error without the call being made at all. Usually there is also some kind of monitoring whether the circuit breaker has tripped.

This simple form of circuit breaker would need an external intervention to reset it when things are well again. An improvement would be to have the breaker itself detect if the underlying cells are working again.