Issue
I don't know much about Webflux / Reactor / Netty. I'm using Spring's WebClient to do all the heavy lifting. But it appears not to work correctly when a server responds back early with an error.
My understanding is when you are POSTing data to a server, the server can respond at any time with an HTTP 4XX error. The client is supposed to stop sending the HTTP body and read that error.
I have a very simply WebClient that POSTs data to a server. It looks like this:
FileResponse resp = client.post().uri(uri)
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.header("Authorization", "Bearer " + authorizationToken)
.accept(MediaType.APPLICATION_JSON)
.bodyValue(data)
.retrieve()
.bodyToMono(FileResponse.class)
.block();
The body can contain a large amount of data (100+KB). Apparently the server looks at the header, validates the authorization token, and only if it's valid, reads the body. If the authorization token is not valid (expired, etc) it immediately responds with an "HTTP 401 Unauthorized" with the response body "{"message": "Invalid user/password"}" while the client is still sending the body. The server then closes the socket which results in the WebClient throwing this:
2022-08-10 15:56:03,474 WARN [reactor.netty.http.client.HttpClientConnect] (reactor-http-nio-1) [id: 0xa7b48bb8, L:/5.6.7.8:51122 - R:dubcleoa030/1.2.3.4:5443] The connection observed an error: java.io.IOException: An existing connection was forcibly closed by the remote host
at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358)
at deployment.bp-global.war//io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at deployment.bp-global.war//io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at deployment.bp-global.war//io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at deployment.bp-global.war//io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at deployment.bp-global.war//io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at deployment.bp-global.war//io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at deployment.bp-global.war//io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at deployment.bp-global.war//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at deployment.bp-global.war//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at deployment.bp-global.war//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at deployment.bp-global.war//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
I've made the same request with curl, and it's handled properly. Curl sees the server's early response, stops sending the body and processes the response from the server. I've chopped out a lot of fluff from the curl output but here is the important stuff...
Trying 1.2.3.4...
* TCP_NODELAY set
* Connected to 1.2.3.4 port 5443 (#0)
> POST /api/folders/file/?path=/out HTTP/1.1
> Host: 1.2.3.4:5443
> User-Agent: curl/7.61.1
> accept-encoding: gzip
> Content-Type: application/octet-stream
> Authorization: Bearer youshallnotpass
> accept: application/json
> Content-Length: 298190
>
< HTTP/1.1 401 Unauthorized
< Server: Cleo Harmony/5.7.0.3 (Linux)
< Date: Wed, 10 Aug 2022 21:59:23 GMT
< Content-Length: 36
< Content-Language: en
< Content-Type: application/json
< Connection: keep-alive
* HTTP error before end of send, stop sending
* Closing connection 0
{"message": "Invalid user/password"}
I'm not sure if the issue is with Spring's WebClient or the underlying reactor-netty stuff. But am I crazy or does it just look broken if the server responds early? If I am correct that it's broken, any thoughts on a work-around?
Thank you! Todd
Solution
I setup a small stand-alone command line Spring Boot program so I could test various aspects of this issue. What I found is that if I send the body from memory (byte[]) the issue occurs. If I send the body from a file resource as shown below, everything works correctly.
Our current very large product is using Spring Boot 2.3.3. My stand-alone test program gave me the ability to quickly upgrade Spring Boot to 2.7.2. Everything works correctly in Spring Boot 2.7.2. So it was definitely a bug that was fixed at some point.
Unfortunately our large project cannot be upgraded to Spring Boot 2.7.2 overnight. It will be a large effort which will require extensive testing from our QA department. So for now, as a work-around, I'll write the body payload to a temporary file so I can get WebClient to read it as a file resource which works in 2.3.3 as shown below. Just in case any other poor developer runs into this and needs a work-around, try this...
InputStreamResource resource = new InputStreamResource(new FileInputStream(tempFilename));
String resp = client.post().uri(uri)
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.header("Authorization", "Bearer youshallnotpass")
.accept(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromResource(resource))
.retrieve()
.bodyToMono(String.class)
.block();
EDIT: I started going through the releases to determine when this was fixed. This issues appears to be fixed in Spring Boot 2.3.4 (Netty 4.1.52). I cannot find any reference to this bug in the release notes of Spring Boot or Netty. So I'm not sure where the issue was. Perhaps it was in a deeper dependency. But it was definitely fixed with Spring Boot 2.3.4.
Answered By - Todd Johnson
Answer Checked By - Marie Seifert (JavaFixing Admin)
