• How many services are written in each language?
• Which services have security vulnerabilities or outdated dependencies?
• What upstream and downstream collaborators use Service A?
• Which services are production-critical? Which are spikes and experiments, or less important to critical application paths?

• Inclusive — Everyone who can do it, should do it, including VPs and directors.
• Fair — On-call work should be remunerated in addition to normal working hours.
• Sustainable — Enough engineers should be in a rotation to avoid burnout and avoid disruption to work-life balance or day-to-work in the office.
• Reflective — Your team should constantly review alerts and pages to ensure only alerts that matter wake someone up.

Increment's on call blogpost

1. Grouping by function

• Unclear ownership
• Lack of autonomy
• No long-term responsibility
• Risk of silos

2. Grouping across functions

• Aligning teams with business value will be reflected in the application developed; the teams will build services that explicitly implement business capabilities.
• Individual services will have clear ownership.
• Service architecture will reflect low coupling and high cohesiveness of teams.
• Functional specialists in different teams can collaborate informally to develop shared pr...

1. Ownership

• A team might own multiple services
• 1:n ownership can lead to conflict about technical choices and make accountability unclear

2. Autonomy

• Important to scale
• Self-forming teams

3. End-to-end responsibility

• Devops

1. Timestamps
2. Identifiers
3. Source
4. Level or category

1. Latency
2. Errors
3. Traffic
4. Saturation

• Metrics
• Logs
• Traces

1. Rolling deploy — You progressively take old instances (version N) out of service while you bring up new instances (version N+1), ensuring that you maintain a minimum percentage of capacity during deployment.
2. Canaries — You add a single new instance into service to test the reliability of version N+1 before continuing with a full rollout. This pattern provides an added measure of safety beyond a normal rolling deploy.
3. Blue-green deploys — You create a parallel group of services (the green set), running the new version of ...

1. A deployment target, or runtime platform, where services are run, such as virtual machines (Ideally, engineers can use an API to configure, deploy, and update service configuration. You also could call this API the control pane, as shown in the figure.)
2. Runtime management, such as auto-healing and autoscaling, that allows the service environment to respond dynamically to failure or changes in load without human intervention (For example, if a service instance fails, it should automatically be replaced.)
3. Logging and monitoring to obs...

• Logging
• Configuration fetching
• Metrics collection
• Data store setup
• Health checks
• Service registry and discovery
• The chosen transport-related boilerplate (AMQP, HTTP)

1. Retries

• Exponential backoff
  1. Use jitter
• Considerations
  1. Always limit the total number of retries.
  2. Use exponential back-off with jitter to smoothly distribute retry requests and avoid compounding load.
  3. Consider which error conditions should trigger a retry and, therefore, which retries are unlikely to, or will never, succeed.

2. Fallbacks

• Graceful degradation
• Caching
• Functional redundancy
• Stubbed data

3. Timeou...

1. Making it easier to onboard team members
2. Getting a good understanding of the code structure and concerns regarding the tech stack that an engineering team uses
3. Limiting the scope of experimentation for production systems as the team builds common knowledge, even if not always in the same tech stack
4. Helping to adhere to best practices

1. Load balancing and service health

• Readiness
• Liveness

2. Rate limits

• Back-pressure via headers

3. Validating reliability and fault tolerance

• Load testing ([The art of scalability](The
  Art of Scalability))
  1. Load test as part of the delivery pipeline
  2. Exploratory load testing to identify limits and test assumptions
• Chaos testing
  1. Chaos toolkit
  2. Principles
  • Define a measurable steady state of normal system ...