When a nginx reverse proxy complains about upstreams sending too big headers, tweaking the buffers responsibly can help to prevent this issue.
Example log message:
upstream sent too big header while reading response header from upstream, client: 192.0.2.100, server: localhost, request: "GET /index.html HTTP/1.1", upstream: "http://198.51.100.123:80/index.html", host: "192.0.2.10:80"
This behaviour was caused by an application that transforms parts of the query from the URL into a response header. If the query in the HTTP request was stuffed with a lot of fuzzy data, the response header rendered too big for the Nginx proxy to handle, resulting in a HTTP 503.
Due to the error, Nginx marks the upstream server as down for as long as
fail_timeout is set to. When enough requests of this kind were sent, all backends were marked down.
So let's increase the size allowed for the headers and we are good, are we? But what parameter do we have to tune and which valued do we use?
The size of the overall proxy buffer is set by
proxy_buffers. By default Nginx uses eight memory pages. This defaults to
8 * 4k = 32k on a system with a page size of 4k.
The buffer for the headers are set by
proxy_buffer_size and count against the overall buffer size. The default is one page.
To limit the amount of buffer which is used to send data to the client is
proxy_busy_buffer_size set to determine this limit. The default are two pages. This value includes the header buffers.
There are two rules affecting the min and max values of
proxy_buffers minus one page
In Nginx the maximum length of the URI is determined by the size of one
large_client_header_buffers since there is stored one line per request in this buffers. The default here is 8k. Larger request are rejected with 313 Request URI Too Large
Since the maximum URI is 8k maximum, the response headers with the information from the URI can be 8k plus the other response headers. Adding a safety margin of 8k to the header buffer enables buffering of a whole line from the request with enough space to fit the regular response headers in.
Here are the required changes on a system with 4k pagesize:
proxy_busy_buffer_size by two pages
proxy_buffers 10 4k;