Calling an outside service

Your module calls a carrier, a rates provider, an invoicing API. It works for six months. Then one day the service on the other side stops answering: it does not refuse, it does not answer.

Without a guard, here is what happens. Every call waits. The workers draining the queue all wait. The web requests that trigger the call wait too. A few minutes later your instance has stopped answering as well, for a failure that was not yours.

That is the project’s interdiction #18, and it is short: every outbound call has a timeout and a circuit breaker.

The timeout

from kewos.resilience import DEFAULT_HTTP_TIMEOUT_SECONDS

response = httpx.get(url, timeout=DEFAULT_HTTP_TIMEOUT_SECONDS)

The kernel’s values, so nobody reinvents them per call:

What is being waited on	Timeout
an outbound HTTP call	30 s
a database query	10 s
a background job	300 s

An HTTP client without a timeout waits forever. That is the default in most libraries, and it is the first link in the failure described above.

The circuit breaker

The timeout bounds one call. The breaker stops calling a service known to be down.

from kewos.resilience import CircuitBreaker, CircuitOpenError

# One per dependency, built ONCE at module boot: it is stateful, and that is
# the whole point.
RATES = CircuitBreaker(name="carrier.rates", failure_threshold=3, timeout_seconds=30)

def rate(parcel):
    try:
        return RATES.call(http_call, parcel)
    except CircuitOpenError:
        return None          # the service is known down: do not even go

Run, against a service that fails every time:

call 1: went out, failed -> the service does not answer (state closed, 1 failure)
call 2: went out, failed -> the service does not answer (state closed, 2 failures)
call 3: went out, failed -> the service does not answer (state open, 3 failures)
call 4: refused BEFORE going out -> CircuitOpenError: circuit breaker 'carrier.rates' is open
call 5: refused BEFORE going out -> CircuitOpenError: circuit breaker 'carrier.rates' is open
calls that actually went out: 3

Three calls went out, two were refused without leaving. That is the difference between a slow dependency and a slow instance: on the fourth call your code answers immediately.

And it closes itself:

after 2 failures, state: open
31 s later, a probe call goes through: answer
state after the successful probe: closed

Past the recovery timeout the breaker lets one probe through. If it succeeds the breaker closes; if it fails it reopens for another round. Nobody rearms anything by hand.

The retry, and when not to

from kewos.resilience import retry_with_backoff

result = retry_with_backoff(call, attempts=3, retry_on=(TimeoutError,))

retry_with_backoff -> answer in 3 attempts

The backoff is exponential with jitter, so your workers do not all retry on the same second.

Never retry an operation that is not idempotent

A GET for a rate, yes. A POST that creates a payment, no: the timeout may have expired while the other side had already taken the money, and the retry takes it a second time.

For those calls the answer is an idempotency key handed to the provider, not a blind retry.

What it does not excuse

An open breaker is not a silent failure. The user has to see that the service is degraded (interdiction #25), and the module has to decide what to do without it: an estimated rate, a queue, a screen that says so. The breaker gives you back control fast so that you have that choice; it does not make it for you.

And the calls you do not make yourself

A call to an AI model goes through kewos/ai, an email through the queue, a document read through kewos/ocr. Those facades already carry their timeout and their breaker: one more reason to go through them rather than calling a provider directly.