[FIXED] ConnectionPoolTimeoutException at caller [spring web applicaiton] logs but no logs at server [springboot]

Issue

I have one issue in PRD. we recently released a springboot application and it has REST API exposed. Mobile/web APP call a legacy spring application which is in spring [not sprintboot] and it is a web applicationwhich then routes and makes a call to the these failing apis in new springboot. We are seeing timeout exception for these apis only . there are lots of other OUTBOUND api calls made from spring legacy web application to other applications eg : login API [which has apis heavy traffic but these legacy apis work well and call other legacy applications. There are no exception/error in logs in springboot application which has these REST API exposed. Infact we only see timeout in spring web application -meaning connection is exhausted but that does not explain why other apis OUTBOUND call are not failing which use same wrapper HTTPClient. Those which fail with timeout dont have request logs in springboot [ obviously because they dont leave spring web application tomcat JVM and die there due to timeout ] So if we say connection pool is exhausted, the other heavey traffic OUTBOUnd calls should also face same issue but we dont see that. All API call OUTWARD use HTTPCLient [apache.] Not clear what is causing issue. I also explicitly defined below in new springboot for server side [I just did it to see if that makes difference but in vain]:

server:
  tomcat:
    connection-timeout: 10s
    max-connections: 20000
    max-threads: 500
    min-spare-threads: 10

tomcat Log at spring web applicaiton [caller]:

org.apache.http.conn.ConnectionPoolTimeoutException
org.apache.http.conn.ConnectionPoolTimeoutException.Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
at 

Any inputs?

Code snippet of Wrapper HTTPClient :

 SSLContext sslContext = SSLContexts.createDefault();
        HostnameVerifier hostnameVerifier = new DefaultHostnameVerifier();
        SSLConnectionSocketFactory secureSSLConnectionSocketFactory = new SSLConnectionSocketFactory(
                sslContext,
                sslProtocolsArray,
                ciphersArray,
                hostnameVerifier);

        ConnectionSocketFactory nonSecureConnectionSocketFactory = PlainConnectionSocketFactory.getSocketFactory();

        Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder
                .<ConnectionSocketFactory>create()
                .register("https", secureSSLConnectionSocketFactory)
                .register("http", nonSecureConnectionSocketFactory)
                .build();

        securePoolingConnectionManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
        securePoolingConnectionManager.setMaxTotal(this.connectionMgrMaxTotalSecure);
        securePoolingConnectionManager.setDefaultMaxPerRoute(this.connectionMgrMaxPerRouteSecure);

        SocketConfig secureSocketConfig = SocketConfig
                .custom()
                .setSoKeepAlive(true)
                .setTcpNoDelay(true)
                .build();

        secureHttpsClient = HttpClients
                .custom()
                .setSSLSocketFactory(secureSSLConnectionSocketFactory)
                .setConnectionManager(securePoolingConnectionManager)
                .setDefaultRequestConfig(secureRequestConfig)
                .setDefaultSocketConfig(secureSocketConfig)
                .disableAutomaticRetries()
                .build();

Stacktrace after above is just failing at wrapper HTTPClient method where call is invoked :

   protected String execute(HttpClient httpclient, HttpRequestBase http) throws IOException {
        String result;
        ResponseHandler<String> responseHandler = new BasicResponseHandler();
        result = httpclient.execute(http, responseHandler);
        return result;
    }

Solution

So I have to dig in another wrapper which was also using this HTTP pool and was being used in our legacy which was leaking. Closing this. Fortunately there was pool statistics api exposed so that I can see leased connection count which confirmed leaking. Since this second wrapper was used rerely and we had used in this release this was suspect and removing it solved the issue. It is another matter to dig that wrapper and find out how the pool was handled but the cause was caught.

Answered By - Vivek Misra