How to build reliable communication

  1. Retries
  • Exponential backoff
    1. Use jitter Show archive.org snapshot
  • Considerations
    1. Always limit the total number of retries.
    2. Use exponential back-off with jitter to smoothly distribute retry requests and avoid compounding load.
    3. Consider which error conditions should trigger a retry and, therefore, which retries are unlikely to, or will never, succeed.
  1. Fallbacks
  • Graceful degradation
  • Caching
  • Functional redundancy
  • Stubbed data
  1. Timeouts
  • If your service interactions are over HTTP, you could propagate deadlines using a custom HTTP header, such as X-Deadline: 1000, passing that value to set read timeout values on subsequent HTTP client calls. Many RPC frameworks, such as gRPC, explicitly implement mechanisms for propagating deadlines within a request context.
  1. Circuit breakers
  2. Asynchronous communication
Sanjay Over 4 years ago