others-how to solve to solve the `13: Permission denied` problem when trying to connect nginx as the grpc proxy server?

Aug 18, 2023

1. Purpose

In this post, I will show you how to solve to solve the 13: Permission denied problem when trying to connect nginx as the grpc proxy server.

2. Problem and Solution

2.1 Problem

when trying to test a grpc service behind a nginx server, I got this error:

➜  .vim_runtime git:(master) grpcurl -insecure  -d '{"helloFrom":"cowrie"}' 10.2.3.21:15051 hello.HelloService/sayHello
Error invoking method "hello.HelloService/sayHello": rpc error: code = Unavailable desc = failed to query for service descriptor "hello.HelloService": unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html"

The nginx server error message:

2023/08/18 10:16:05 [crit] 17809#17809: *4 connect() to [::1]:8500 failed (13: Permission denied) while connecting to upstream, client: 10.3.2.238, server: 10.2.3.21, request: "POST /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0", upstream: "grpc://[::1]:8500", host: "10.2.3.21:15051"
2023/08/18 10:16:05 [crit] 17809#17809: *4 connect() to 127.0.0.1:8500 failed (13: Permission denied) while connecting to upstream, client: 10.3.2.238, server: 10.2.3.21, request: "POST /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0", upstream: "grpc://127.0.0.1:8500", host: "10.2.3.21:15051"
2023/08/18 10:16:05 [error] 17809#17809: *4 no live upstreams while connecting to upstream, client: 10.3.2.238, server: 10.2.3.21, request: "POST /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0", upstream: "grpc://localhost", host: "10.2.3.21:15051"

You can see that the core error message is:

connect() to [::1]:8500 failed (13: Permission denied) while connecting to upstream

However, if I test the grpc directly (bypass nginx), it’s working

➜  .vim_runtime git:(master) grpcurl -plaintext -d '{"helloFrom":"cowrie"}' 10.2.3.21:8500 hello.HelloService/sayHello
{
  "hello_result": "hi,cowrie"
}

So the problem is on the nginx server!

2.2 Environment

Nginx version:

[root@local ~]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.1.1k  FIPS 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

The nginx.conf

    server {
        listen 15051 ssl http2;
        server_name 10.2.3.21;

        ssl on;
        ssl_certificate /etc/nginx/conf.d/certs/example.crt;
        ssl_certificate_key /etc/nginx/conf.d/certs/example.key;

        ssl_session_timeout 5m;

        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers   on;

        location / {
                grpc_pass grpc://localhost:8500;
        }
    }

2.3 Solution and Reason

solution:

setsebool -P httpd_can_network_connect 1

because:

Each boolean value corresponds to a specific SELinux rule or policy. For example, the boolean value httpd_can_network_connect controls whether the Apache web server/nginx is allowed to make network connections. Boolean values can be set using the setsebool command in Linux.

After restarting nginx, now it’s working fine.

3. Summary

In this post, I demonstrated how to solve the 13: Permission denied problem when trying to use nginx to proxy grpc service, the key point is to check if you have enabled SELinux and also check if you have allowed nginx to make connections. That’s it, thanks for your reading.