When a nginx reverse proxy complains about upstreams sending too big headers, tweaking the buffers responsibly can help to prevent this issue.

Do you need DevOps-experts?

Your development team has a full backlog? No time for infrastructure architecture? Our DevOps team is ready to support you!

• We build reliable cloud solutions with Infrastructure as code
• We are experts in security, Linux and databases
• We support your dev team to perform

Read more Show archive.org snapshot

Example log message:

upstream sent too big header while reading response header from upstream, client: 192.0.2.100, server: localhost, request: "GET /index.html HTTP/1.1", upstream: "http://198.51.100.123:80/index.html", host: "192.0.2.10:80"

The cause

This behaviour was caused by an application that transforms parts of the query from the URL into a response header. If the query in the HTTP request was stuffed with a lot of fuzzy data, the response header rendered too big for the Nginx proxy to handle, resulting in a HTTP 503.

Due to the error, Nginx marks the upstream server as down for as long as fail_timeout is set to. When enough requests of this kind were sent, all backends were marked down.

So let's increase the size allowed for the headers and we are good, are we? But what parameter do we have to tune and which valued do we use?

Buffer size

The size of the overall proxy buffer is set by proxy_buffers. By default Nginx uses eight memory pages. This defaults to 8 * 4k = 32k on a system with a page size of 4k.

Header buffer size

The buffer for the headers are set by proxy_buffer_size and count against the overall buffer size. The default is one page.

Maximum buffer usage

To limit the amount of buffer which is used to send data to the client is proxy_busy_buffer_size set to determine this limit. The default are two pages. This value includes the header buffers.

There are two rules affecting the min and max values of proxy_busy_buffer_size:

• the minimum equals to proxy_buffer_size
• the maximum is proxy_buffers minus one page

Maximum URI size

In Nginx the maximum length of the URI is determined by the size of one large_client_header_buffers since there is stored one line per request in this buffers. The default here is 8k. Larger request are rejected with 313 Request URI Too Large

The solution

Since the maximum URI is 8k maximum, the response headers with the information from the URI can be 8k plus the other response headers. Adding a safety margin of 8k to the header buffer enables buffering of a whole line from the request with enough space to fit the regular response headers in.

Here are the required changes on a system with 4k pagesize:

• add two pages to proxy_buffers
• add two pages to proxy_buffer_size
• shift proxy_busy_buffer_size by two pages

proxy_buffers 10 4k;
proxy_buffer_size 12k;
proxy_busy_buffers_size 16k;