Posted about 2 years ago. Visible to the public. Draft.
How to build reliable communication
- Exponential backoff
- Always limit the total number of retries.
- Use exponential back-off with jitter to smoothly distribute retry requests and avoid compounding load.
- Consider which error conditions should trigger a retry and, therefore, which retries are unlikely to, or will never, succeed.
- Graceful degradation
- Functional redundancy
- Stubbed data
- If your service interactions are over HTTP, you could propagate deadlines using a custom HTTP header, such as X-Deadline: 1000, passing that value to set read timeout values on subsequent HTTP client calls. Many RPC frameworks, such as gRPC, explicitly implement mechanisms for propagating deadlines within a request context.
- Circuit breakers
- Asynchronous communication