[FIXED] Connections leaking with state CLOSE_WAIT with HttpClient

Issue

We are using JDK11 java.net.http HTTP client to get data from an API. After we receive the response the connections remain opened in our server with TCP state CLOSE_WAIT, which means the client must close the connection.

From href="https://www.rfc-editor.org/rfc/rfc793#section-3.2" rel="nofollow noreferrer">RFC 793 terminology:

CLOSE-WAIT - represents waiting for a connection termination request from the local user.

This is our client code which runs on WildFly 16 running on Java 12 as a stateless REST API. We don't understand why this is happening.

import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpClient.Version;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;

public class SocketSandbox {

    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newBuilder().version(Version.HTTP_1_1).build();
        try (var listener = new ServerSocket(59090)) {
            System.out.println("Server is running...");
            while (true) {
                try (var socket = listener.accept()) {
                    HttpRequest request = HttpRequest
                            .newBuilder(URI.create("<remote_URL>"))
                            .header("content-type", "application/json").build();
                    HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
                    var out = new PrintWriter(socket.getOutputStream(), true);
                    out.println(String.format("Response HTTP status: %s", response.statusCode()));
                }
            }
        }
    }

}

We get the "status code" meaning the http response was processed.

When using the same code to call other endpoints connections are fine. It seems to be a particular issue with the remote API we are calling, but still we don't understand why the Java HTTP client is keeping connections opened.

We tried both Windows and Linux machines, and even standalone outside of WildfFly, but the same result happens. After each request, even doing it from our stateless client and receiving the response, each one is left as CLOSE_WAIT and never close.

Connections will disappear if we shutdown the Java process.

Headers that are sent by the HTTP client:

connection: 'Upgrade, HTTP2-Settings','content-length': '0',
host: 'localhost:3000', 'http2-settings': 'AAEAAEAAAAIAAAABAAMAAABkAAQBAAAAAAUAAEAA',
upgrade: 'h2c',
user-agent': 'Java-http-client/12'

Server returns response with header: Connection: close

Update (1)

We tried to fine-tune the pool parameters in implementation class jdk.internal.net.http.ConnectionPool.

It did not solve the problem.

System.setProperty("jdk.httpclient.keepalive.timeout", "5"); // seconds
System.setProperty("jdk.httpclient.connectionPoolSize", "1");

Update (2)

With Apache HTTP the connections l getting left in CLOSE_WAIT state for about 90 seconds, but it is able to the connections after that time.

Calling method HttpGet.releaseConnection() force the connection close immediately.

HttpClient client = HttpClients.createDefault();
HttpGet get = new HttpGet("https://<placeholderdomain>/api/incidents/list");
get.addHeader("content-type", "application/json");
HttpResponse response = client.execute(get);

// This right here did the trick
get.releaseConnection();

return response.getStatusLine().getStatusCode();

And with OkHttp client it worked as expected out of the box, no connections stuck.

OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
        .url("https://<placeholderdomain>/grb/sb/incidentes/listar")
        .header("content-type", "application/json").build();
Response response = client.newCall(request).execute();
return response.body().string();

We are still trying to find how to make it work in java-http-client so that we don't have to rewrite the code.

Solution

Submitted and confirmed as a bug in the implementation.

https://bugs.openjdk.java.net/browse/JDK-8221395

Update

Check the JIRA issue, it's fixed in JDK 13, and backported to 11.0.6. (Not sure about 12)

Answered By - Evandro Pomatti
Answer Checked By - Dawn Plyler (JavaFixing Volunteer)