others-How to solve 502 bad gateway error when using proxy_pass in Nginx and Tomcat?

1. Introduction

In this post, I would show how to solve 502 bad gateway error when using proxy_pass in Nginx.

2. Environment

  • Nginx 1.x

3. The nginx.conf

We can define an upstream in nginx to reverse proxy local tomcat service like this:

The service layout: image desc

http {
    ...
    # define an upstream to be reversely proxied
    # There are two local tomcat services that are listening to port 8081 and 8082
    upstream service {
        server localhost:8081; # tomcat 1
        server localhost:8082; # tomcat 2
    }
    ...
    #define a server to proxy the services
    server{
        listen       80;
        ...
        location /myservice {
                proxy_pass http://service;
        }
    }
}

When user nagivate to http://xxx.com/myservice, then it would be reverse proxied to localhost’s tomcat instances(8081 and 8082).

4. The problem

When your business grows, the concurrent http requests would burst in a short time, sometimes, your users would get a error like this when requesting your services:

<html>
    <head>
        <title>502 Bad Gateway</title>
    </head>
    <body bgcolor="white">
        <center>
            <h1>502 Bad Gateway</h1>
        </center>
        <hr>
        <center>nginx</center>
    </body>
</html>

5. How to solve this 502 bad gateway problem?

5.0 Why did this happen?

Because you did not use the keep-alive feature of http, every request create a new connection, then there are too many connections in your server.

Default http 1.0 connections and requests: image desc

You can see that every request create a new connection,so your server would suffer from the surge requests.

The http 1.1 connections and requests: image desc

You can see that there are only a few connections between the nginx and local service, many of them are reused between http requests.

5.1 Try to use http keep alive feature

Change your server settings in nginx.conf, try to activate http 1.1 keep-alive features like this:

    #define a server to proxy the services
    server{
        listen       80;
        ...
        location /myservice {
                proxy_http_version 1.1; #use http 1.1 to connect local service
                proxy_set_header Connection ""; #remove client connection headers

                proxy_pass http://service;
        }
    }

The explaination:

  • “proxy_http_version 1.1” means use http 1.1, because the keep-alive feature is a default feature in http 1.1 protocol
  • “proxy_set_header Connection “”” means clear the client’s connection headers, use http 1.1 header

5.2 Set the maximum number of idle keep-alive connections in upstream

Change your upstream settings like this:

http {
    ...
    # define an upstream to be reversely proxied
    # There are two local tomcat services that are listening to port 8081 and 8082
    upstream service {
        # The keepalive parameter sets the maximum number of idle keepalive connections
        # to upstream servers that are preserved in the cache of each worker process. When
        # this number is exceeded, the least recently used connections are closed.
        keepalive 50;

        server localhost:8081; # tomcat 1
        server localhost:8082; # tomcat 2
    }
    ...
}

5.3 Decrease the keep-alive timeout

http {
    ...
    #change the default 15 seconds to 5 seconds to decrease the load of the server
    keepalive_timeout  5;
    ...
}

6. The final nginx.conf

http {
    #change the default 15 seconds to 5 seconds to decrease the load of the server
    keepalive_timeout  5;
    ...
    # define an upstream to be reversely proxied
    # There are two local tomcat services that are listening to port 8081 and 8082
    upstream service {
        # The keepalive parameter sets the maximum number of idle keepalive connections
        # to upstream servers that are preserved in the cache of each worker process. When
        # this number is exceeded, the least recently used connections are closed.
        keepalive 50;

        server localhost:8081; # tomcat 1
        server localhost:8082; # tomcat 2
    }
    ...
    #define a server to proxy the services
    server{
        listen       80;
        ...
        location /myservice {
                proxy_http_version 1.1; #use http 1.1 to connect local service
                proxy_set_header Connection ""; #remove client connection headers

                proxy_pass http://service;
        }
    }
}

6. Summary

You should always try to use the keep-alive feature of http, it would help you to obtain good user experience and good revenue too.