[FIXED] Tomcat 9 Http/2 "Error reading request, ignored"

Issue

I m using tomcat 9.0.5(jdk-9.0.4+11) to serve my requests which also have Http/2 enabled for persistent connection and my client also uses the same version of java, I need to open a persistent connection to the server from client and reuse indefinitely. I can successfully connect to my server from client and send request(Post with requestBody) but after 3 requests(exactly 3) the server throws as exception as follows,

org.apache.coyote.AbstractProtocol$ConnectionHandler.process Error reading request, ignored java.lang.NullPointerException at org.apache.coyote.http2.HpackDecoder.handleIndex(HpackDecoder.java:270) at org.apache.coyote.http2.HpackDecoder.decode(HpackDecoder.java:111) at org.apache.coyote.http2.Http2Parser.readHeaderPayload(Http2Parser.java:418) at org.apache.coyote.http2.Http2Parser.readHeadersFrame(Http2Parser.java:252) at org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:97) at org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) at org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:313) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:754) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1376) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:844)

So a connection is open for only three request after which server closes the connection upon this exception.

p.s: I m using httpclient-5(beta) for establishing persistent connection from my client. If the queryParam/requestBody is static content the connection is kept persistent. but, if the queryParam/requestBody have some dynamic content(like timestamps) the connection is closed with the above error I don't know the relation between the data in the post body and the tcp connection

I can't post my request body as it is against policy I will post after masking all my request params, my request will look something like this,

String postBody = "<?xml version=\"1.0\" standalone=\"no\"?>\n" + 
                "<XX xxx=\"1000000005011\" xxx=\"true\" xxx=\"false\" xxx=\"0\" xxx=\"P|TA|D|J|M\" xxx=\"\" xxx=\"\" xxx=\"false\" xxx=\"false\" xxx=\"1\" xxx=\"true\" xxx=\"false\" xxx=\"xxx\" xxx=\"8265\"  xxx=\"true\" avgDnsTime=\"null\" xxx=\"false\" xxx=\"EVAL_USER\" xxx=\"1000000000011\" xxx=\"2\" xxx=\"true\" xxx=\"http://localhost:8080/app/xxx\" xxx=\"true\" xxx=\"false\" xxx=\"av_xxx\" ut=\"1528453675150\" xxx=\"URL\" xxx=\"70\" time=\""+(System.currentTimeMillis()+"")+"\" xxx=\"1440\" xxx=\"48\" xxx=\"1\" xxx=\"1\" postUrl=\"xxx.com:8443\">\n" + 
                "<XXX xxx=\"\" xxx=\"\" xxx=\"false\" xxx=\"\" xxx=\"1000000005011\" xxx=\"60\" xxx=\"\" xxx=\"127.0.0.1\" xxx=\"\" xxx=\"false\" xxx=\"\" xxx=\"default\" xxx=\"\" xxx=\"\" xxx=\"\" xxx=\"false\" xxx=\"false\" xxx=\"false\" xxx=\"http://www.example.com\" xxx=\"\" xxx=\"\" xxx=\"false\" m=\"G\" xxx=\"0\" xxx=\"\" xxx=\"\" t=\"30\" xxx=\"English\"><XD></XD><XH xxx=\"\" xxx=\"_sep_\" xxx=\"\"/><XI xxx=\"\" xxx=\"\"/>\n" + 
                "</XXX>\n" + 
                "</XX>";

If anyone have any idea about this exception kindly help

TIA

Solution

As discussed in the comments

HPACK is the HTTP Header Compression algorithm used in HTTP/2. So the error messages suggest a problem with your HTTP headers.

HTTP/1 was very forgiving with malformed HTTP headers, but HTTP/2 is much more strict. Extra newlines, open quotes, extra colons...etc. can all cause problems.

Additionally, under HTTP/2 all the request methods and params are consider HTTP Headers.

So, this in HTTP/1:

GET /page.html HTTP/1.1
Header1: Header1Value
Header2: Header2Value

becomes this in HTTP/2:

:method: GET
:path: /page.html
Header1: Header1Value
Header2: Header2Value

So check all your headers (including your path) for malformed requests.

In your case it was a badly encoded query Param in our :path: header.

Answered By - Barry Pollard