Implementation of Nginx's passive retry mechanism

When we use Nginx to do load balancing through reverse proxy, if an error occurs or timeout occurs in one of the proxy services, we usually hope that Nginx will automatically retry other services to achieve high service availability. In fact, Nginx itself has an error retry mechanism by default, and it can be passedproxy_next_upstreamCustomize the configuration.

Nginx byproxy_next_upstreamParameters to define what situations will be considered fails, which triggers the failed retry mechanism.

Fails can be divided into two categories:

• Default error, including error, timeout
• Error in selection definition, including invalid_header and various exceptions http status code errors, etc.

Default error

AppearerrorIn scenarios, common are the service of the upstream server restarts, stops, or fails to provide normal services due to abnormal crashes. andtimeoutThe situation is that the corresponding timeout configuration is achieved during the proxy request process, mainly including:

• proxy_connect_timeout, establish three handshake time
• proxy_read_timeout, After establishing a connection, wait for the upstream server to respond and process the request
• proxy_send_timeout, the interval between data return (note that it is not the time to send data)

Error in selection definition

The exception status code part (that is, 4xx, 5xx errors). The upstream server returns an empty response or an illegal response header

invalid_header: a server returned an empty or invalid response;

Its default value isproxy_next_upstream error timeout, that is, network errors and timeouts occur, and other servers will be retryed. By default, the service returns a 500 status code and will not retry

Command configuration

proxy_next_upstream

Sets the first time that a server in the upstream server cluster fails to connect to a server, specifying the situations under which the request will be delivered to the next server.

grammar：	proxy_next_upstream error | timeout | invalid_header | http_500 | http_502 | http_503 | http_504 | http_403 | http_404 | http_429 | non_idempotent | off ...;
default：	proxy_next_upstream error timeout;
Use location：	http, ,serverlocation

• error # An error occurs when establishing a connection with the server, passing a request or reading a response header to it;
• timeout # Timeout occurs when establishing a connection with the server, passing a request to it, or reading a response header;
• invalid_header # The server returns an empty or invalid response;
• http_500 # The server returns a response with a code of 500;
• http_502 # The server returns a response with a code of 502;
• http_503 # The server returns a response with code 503;
• http_504 # The server returns the response of code 504;
• http_403 # The server returns a response with code 403;
• http_404 # The server returns a response with a code of 404;
• http_429 # The server returns a response with code 429 (1.11.13);
• non_idempotent # Usually, the request is not passed with the non-idempotent method (POST, LOCK, PATCH) to the next server of whether the request has been sent to the upstream server (1.9.13); enable this option to explicitly allow retry of such requests;
• off # Disable passing requests to the next server.

When the request type is POST, Nginx will not fail to retry by default，If you want the POST request to fail and retry, you need to configure itnon_idempotent。

Configuration example:

Code language: javascript

upstream nginxretry {
    server 127.0.0.1:9030 weight=10;
	server 127.0.0.1:9031 weight=10;
}
server {
    listen 9039;
    location / {
        proxy_pass http://nginxretry;
        proxy_next_upstream error timeout http_500;
    }
}

proxy_next_upstream_timeout

Set the timeout time for retry, no retry will be done after the timeout, and the error will be returned to the user. The default is 0, that is, no restrictions will be made.

proxy_next_upstream_tries

Set the maximum number of retry times. If the number of retry times exceeds, no retry will be tried again. The default is 0, that is, no limit is given (proxy_next_upstream_timeout allows proxy_next_upstream_tries to be retryed within the proxy_next_upstream_tries, including the first time)

Configuration example:

server {
	proxy_next_upstream error timeout;
	proxy_next_upstream_timeout 15s;
	proxy_next_upstream_tries 5;
}

Retry restriction method

The default configuration does not require a retry mechanism, which means that it will try again until it fails.

Nginx provides the following two parameters to control the number of retry times and the retry timeout:

• proxy_next_upstream_tries: Set the number of retry times, default0Indicates that there is no limit, this parameter contains the number of times all requested upstream server, including the sum of all retry after the first time;
• proxy_next_upstream_timeout: Set the maximum timeout for retry, default0Indicates no limit. This parameter refers to the first connection time plus the subsequent retry connection time, and does not include the processing time after connecting to the node.

Restrictions on a single server in upstream

• max_fails: Maximum number of failures (0 is the mark that has always been available, and the health status is not checked)
• fail_timeout: Fail time (when max_fails failed within the fail_timeout time, the service will be activated again after the fail_timeout time is not available)

Configuration Example 1:

upstream httpget {
    server 192.168.111.101:8080 max_fails=5 fail_timeout=10s;
    server 192.168.111.102:8080;
}

Configuration Example 2:

proxy_connect_timeout 3s;
proxy_next_upstream_timeout 6s;
proxy_next_upstream_tries 3;

upstream test {
    server 127.0.0.1:8001 fail_timeout=60s max_fails=2; # Server A
    server 127.0.0.1:8002 fail_timeout=60s max_fails=2; # Server B
    server 127.0.0.1:8003 fail_timeout=60s max_fails=2; # Server C
}

This is the end of this article about the implementation of the passive retry mechanism of Nginx: Nginx upstream. For more information about the passive retry mechanism of Nginx upstream, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!