Issue
Our team recently had some problems with Jenkins Plugin Manager installing / updating plugins. After we fixed some problems with our firewall and proxy rules (we are behind a corporate firewall and a proxy), we still had issues downloading some of the plugin files.
Some plugin files were downloaded and installed successfully, others failed with the following error:
Failed to install git-client
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3057)
at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)
at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)
at java.net.URLConnection.getContentLength(URLConnection.java:485)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1161)
Caused: java.net.SocketTimeoutException: Read timed out
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1177)
Caused: java.io.IOException: Failed to load http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi to /var/lib/jenkins/plugins/git-client.jpi.tmp
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1184)
Caused: java.io.IOException: Failed to download from http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi (redirected to: http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1218)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1766)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2037)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1740)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
at java.lang.Thread.run(Thread.java:748)
To diagnose potential network problems, I tried downloading git-client.hpi directly from the machine hosting the Jenkins server via curl, which was successful:
[root@servername ~]# curl -vk -x http://proxyserver:7689 http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi -o /tmp/git-client.hpi
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to proxy proxyserver port 6789 (#0)
* Trying 10.100.50.100...
* Connected to proxyserver (10.100.50.100) port 6789(#0)
> GET http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ftp-nyc.osuosl.org
> Accept: */*
> Proxy-Connection: Keep-Alive
>
0 0 0 0 0 0 0 0 --:--:-- 0:01:01 --:--:-- 0< HTTP/1.1 200 OK
< Date: Mon, 11 Nov 2019 08:18:55 GMT
< Server: Apache
< Last-Modified: Sat, 02 Nov 2019 12:03:46 GMT
< ETag: "7dd268-5965bdfed67ce"
< Accept-Ranges: bytes
< Content-Type: application/vnd.hp-hpid
< Age: 6915
< Content-Length: 8245864
< Via: 1.1 proxyserver
<
0 8052k 0 0 0 0 0 0 --:--:-- 0:01:01 --:--:-- 0{ [data not shown]
100 8052k 100 8052k 0 0 130k 0 0:01:01 0:01:01 --:--:-- 2132k
* Connection #0 to host proxyserver left intact
Downloading via curl was successful, although I noticed that after successful connection, the download process was stalled at 0% for about a minute. Then it suddenly started, and finished in seconds.
EDIT: this is not an insfrastructure problem, and has nothing to do with INFRA-2393, nor any other problems related to update site infrastructure:
This read timeout was encountered first in October 2019, and could be reproduced in the middle of December 2019. Infrastucture problems tend to get solved a lot faster than that.
I also managed to reproduce the problem at about 2020-01-02 10:30 CET, with /plugins/subversion/2.13.0/subversion.hpi using a 60 sec read timeout setting, long after INFRA-2393 got resolved. I also managed to fix the problem by raising the read timeout setting as described below.
The read timeout only happens within Jenkins. Using the same download link in a browser or in a curl command downloads the affected files. This is because Jenkins has a default read timeout of 60 seconds, while browsers / curl do not.
While diagnosing this earlier, we rearranged our network policy and made Jenkins reach the update sites directly (without using a proxy). In such a context, all the problem downloads worked. This indicates the stalling download is not related to an update site bug, but to our own proxy.
Also, stop asking me to file a report on Jenkins Jira. The very reason I posted this question and my answer is that I got frustrated with trying to find a solution using Jenkins Jira. The material that I found related to my problem there roughly fell into two categories:
Infrastructure issues dating back several years, obviously fixed years ago. In other words: noise.
Genuine Jenkins issues posted with similar symptoms, with dozens / hundreds of comments posted, mostly boiling down to Jenkins staff stating that the problem is on the user's side, and that people should fix their own corporate firewall / proxy / etc. Not a word related to timeout values whatsoever. In other words: a lot more noise.
Again: this is not a problem related to Jenkins update sites, nor is it a Jenkins software bug. If it's related to anything, then it's the lack of documentation. Exactly the thing that SO is meant to cover IMHO.
Solution
The important thing to notice from the above is that this is not a connection timeout but a read timeout. The connection to the download mirror is estabilished, but the client does not receive a single byte for about a minute. (We hypothesised that this happens because our proxy analyses the contents of git-client.hpi before forwarding the bytes to the client that requested the download.)
The curl download is successful because there's no read timeout set in the command. Jenkins obviously uses a 60 second default socket read timeout, and so after there's no traffic for 60 seconds, it considers the connection dead and fails with a SocketException.
I did not find any official documentation on this, but looking at Jenkins source code confirms the theory: the UpdateCenter.PLUGIN_DOWNLOAD_READ_TIMEOUT constant is initialized from a property value, and defaults to 60 in its absence.
The other thing I did not find is a comprehensive documentation of Jenkins' properties (key names, possible values, description, etc.).
The exact property key to be set is: hudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds
AFAIK Jenkins does not have any *.properties file, but accepts property settings on startup as JVM parameters.
I solved this problem by finding the jenkins options file (if you use RHEL, and installed Jenkins from package repository, the exact path is: /etc/sysconfig/jenkins), and appending -Dhudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds=120 to the value of the JENKINS_JAVA_OPTIONS setting.
UPDATE: I found the official Jenkins documentation about Jenkins system properties, which also has a few words about the hudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds property.
UPDATE (2022-07-08):
Beginning with Jenkins 2.332.1 and Jenkins 2.335, the Linux package installers use systemd to manage services. The RPM and deb package installers migrate configuration settings from System V init to systemd overrides.
This means that configuration files formerly used (/etc/sysconfig/jenkins or /etc/default/jenkins, depending on distribution) are now outdated and systemd unit files should be used to set system properties. See Jenkins documentation.
Answered By - Attila Csipak
Answer Checked By - David Marino (JavaFixing Volunteer)
