others-How to solve the `Could not get a resource from the pool` exception when using spring data redis to connect redis?

Jan 12, 2022

1. Purpose

In this post, I would demonstrate how to sovle the below exception when using spring data redis in spring applications.

INFO   | jvm 1    | 2022/01/05 14:22:09 | 2022-01-05 14:22:09.870 DEBUG 20276 --- [nio-9997-exec-2] com.bswen.iface.web.mybotController     : send mybot task to redis ok com.bswen.myapp3.common.domain.SendmybotRequest@314f7de3
INFO   | jvm 1    | 2022/01/05 14:22:11 | 2022-01-05 14:22:11.878 ERROR 20276 --- [nio-9997-exec-2] o.a.c.c.C.[.[.[.[dispatcherServlet]      : Servlet.service() for servlet [dispatcherServlet] in context with path [/iface] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool] with root cause
INFO   | jvm 1    | 2022/01/05 14:22:11 |
INFO   | jvm 1    | 2022/01/05 14:22:11 | java.net.SocketTimeoutException: connect timed out
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at redis.clients.jedis.Connection.connect(Connection.java:158) ~[jedis-2.7.3.jar:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:82) ~[jedis-2.7.3.jar:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1641) ~[jedis-2.7.3.jar:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:85) ~[jedis-2.7.3.jar:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868) ~[commons-pool2-2.4.2.jar:2.4.2]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) ~[commons-pool2-2.4.2.jar:2.4.2]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at redis.clients.util.Pool.getResource(Pool.java:48) ~[jedis-2.7.3.jar:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at redis.clients.jedis.JedisPool.getResource(JedisPool.java:99) ~[jedis-2.7.3.jar:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at redis.clients.jedis.JedisPool.getResource(JedisPool.java:12) ~[jedis-2.7.3.jar:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:155) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:251) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:58) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:178) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.data.redis.core.DefaultHashOperations.increment(DefaultHashOperations.java:74) ~[spring-data-redis-1.6.1.RELEASE.jar:1.6.1.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at com.bswen.iface.web.mybotController.incrCount(mybotController.java:87) ~[myapp2-interface-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at com.bswen.iface.web.mybotController.realProcess(mybotController.java:70) ~[myapp2-interface-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at com.bswen.iface.web.mybotController.save(mybotController.java:55) ~[myapp2-interface-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) ~[na:na]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[tomcat-embed-core-8.0.28.jar:8.0.28]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[tomcat-embed-core-8.0.28.jar:8.0.28]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) ~[tomcat-embed-core-8.0.28.jar:8.0.28]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[tomcat-embed-core-8.0.28.jar:8.0.28]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.0.28.jar:8.0.28]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) ~[tomcat-embed-core-8.0.28.jar:8.0.28]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[tomcat-embed-core-8.0.28.jar:8.0.28]
INFO   | jvm 1    | 2022/01/05 14:22:11 |       at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
Timeout, server 174.137.63.192 not responding.  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]

The core error message is:

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool] with root cause
INFO   | jvm 1    | 2022/01/05 14:22:11 |
INFO   | jvm 1    | 2022/01/05 14:22:11 | java.net.SocketTimeoutException: connect timed out

And this:

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

2. The solution

2.1 Environment

spring boot 1.3.0
spring-data-redis 1.6.1
jedis 2.7.3

2.2 My solution

My solution to this problem is just ajust the network parameters in system.

Open and edit /etc/sysctl.conf, change the following settings.

net.ipv4.tcp_tw_reuse=0
net.ipv4.tcp_tw_recycle=0

What is net.ipv4.tcp_tw_reuse?

ipv4. tcp_tw_reuse. Permits sockets in the time-wait state to be reused for new connections. … This parameter, when set, allows no longer needed and about to be destroyed sockets to be used for new connections.

What is net.ipv4.tcp_tw_recycle?

tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4) Enable fast recycling of TIME-WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).

And then:

sysctl -p

Verify:

sysctl -a|grep tcp_tw_reuse
sysctl -a|grep tcp_tw_recycle

After some days, I found that the problem disappeared.

3. Summary

In this post, I demonstrated how to solve the ‘Could not get a resource from the pool’ exception when using spring data redis. That’s it, thanks for your reading.