Issue
I use Bitbucket(git) server and currently the git clone is randomly getting hung. The same repo sometimes gets cloned and sometimes get hung unexpectedly and then gets timed out. The server memory, CPU utilization, disk space is fine. Please advise what could be other areas that needs attention to find the root cause.This is happening on 5 different Jenkins slave so it is not restricted to a single server.
Following is the error log:
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "/usr/bin/git -c core.askpass=true fetch --tags --progress http://bitbucket.net/project.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: remote: Counting objects: 2506, done.
stdout:
stderr: remote: Counting objects: 2506, done.
remote: Compressing objects: 0% (1/2405)
remote: Compressing objects: 1% (25/2405)
remote: Compressing objects: 2% (49/2405)
remote: Compressing objects: 3% (73/2405)
remote: Compressing objects: 4% (97/2405)
remote: Compressing objects: 5% (121/2405)
remote: Compressing objects: 6% (145/2405)
remote: Compressing objects: 7% (169/2405)
remote: Compressing objects: 8% (193/2405)
remote: Compressing objects: 9% (217/2405)
remote: Compressing objects: 10% (241/2405)
remote: Compressing objects: 11% (265/2405)
remote: Compressing objects: 12% (289/2405)
remote: Compressing objects: 13% (313/2405)
remote: Compressing objects: 14% (337/2405)
remote: Compressing objects: 15% (361/2405)
remote: Compressing objects: 16% (385/2405)
remote: Compressing objects: 17% (409/2405)
remote: Compressing objects: 18% (433/2405)
remote: Compressing objects: 19% (457/2405)
remote: Compressing objects: 20% (481/2405)
remote: Compressing objects: 21% (506/2405)
remote: Compressing objects: 22% (530/2405)
remote: Compressing objects: 23% (554/2405)
remote: Compressing objects: 24% (578/2405)
remote: Compressing objects: 25% (602/2405)
remote: Compressing objects: 26% (626/2405)
remote: Compressing objects: 27% (650/2405)
remote: Compressing objects: 28% (674/2405)
remote: Compressing objects: 29% (698/2405)
remote: Compressing objects: 30% (722/2405)
remote: Compressing objects: 31% (746/2405)
remote: Compressing objects: 32% (770/2405)
remote: Compressing objects: 33% (794/2405)
remote: Compressing objects: 34% (818/2405)
remote: Compressing objects: 35% (842/2405)
remote: Compressing objects: 36% (866/2405)
remote: Compressing objects: 37% (890/2405)
remote: Compressing objects: 38% (914/2405)
remote: Compressing objects: 39% (938/2405)
remote: Compressing objects: 40% (962/2405)
remote: Compressing objects: 41% (987/2405)
remote: Compressing objects: 42% (1011/2405)
remote: Compressing objects: 43% (1035/2405)
remote: Compressing objects: 44% (1059/2405)
remote: Compressing objects: 45% (1083/2405)
remote: Compressing objects: 46% (1107/2405)
remote: Compressing objects: 47% (1131/2405)
remote: Compressing objects: 48% (1155/2405)
remote: Compressing objects: 49% (1179/2405)
remote: Compressing objects: 50% (1203/2405)
remote: Compressing objects: 51% (1227/2405)
remote: Compressing objects: 52% (1251/2405)
remote: Compressing objects: 53% (1275/2405)
remote: Compressing objects: 54% (1299/2405)
remote: Compressing objects: 55% (1323/2405)
remote: Compressing objects: 56% (1347/2405)
remote: Compressing objects: 57% (1371/2405)
remote: Compressing objects: 58% (1395/2405)
remote: Compressing objects: 59% (1419/2405)
remote: Compressing objects: 60% (1443/2405)
remote: Compressing objects: 61% (1468/2405)
remote: Compressing objects: 62% (1492/2405)
remote: Compressing objects: 63% (1516/2405)
remote: Compressing objects: 64% (1540/2405)
remote: Compressing objects: 65% (1564/2405)
remote: Compressing objects: 66% (1588/2405)
remote: Compressing objects: 67% (1612/2405)
remote: Compressing objects: 68% (1636/2405)
remote: Compressing objects: 69% (1660/2405)
remote: Compressing objects: 70% (1684/2405)
remote: Compressing objects: 71% (1708/2405)
remote: Compressing objects: 72% (1732/2405)
remote: Compressing objects: 73% (1756/2405)
remote: Compressing objects: 74% (1780/2405)
remote: Compressing objects: 75% (1804/2405)
remote: Compressing objects: 76% (1828/2405)
remote: Compressing objects: 77% (1852/2405)
remote: Compressing objects: 78% (1876/2405)
remote: Compressing objects: 79% (1900/2405)
remote: Compressing objects: 80% (1924/2405)
remote: Compressing objects: 81% (1949/2405)
remote: Compressing objects: 82% (1973/2405)
remote: Compressing objects: 83% (1997/2405)
remote: Compressing objects: 84% (2021/2405)
remote: Compressing objects: 85% (2045/2405)
remote: Compressing objects: 86% (2069/2405)
remote: Compressing objects: 87% (2093/2405)
remote: Compressing objects: 88% (2117/2405)
remote: Compressing objects: 89% (2141/2405)
remote: Compressing objects: 90% (2165/2405)
remote: Compressing objects: 91% (2189/2405)
remote: Compressing objects: 92% (2213/2405)
remote: Compressing objects: 93% (2237/2405)
remote: Compressing objects: 94% (2261/2405)
remote: Compressing objects: 94% (2278/2405)
remote: Compressing objects: 95% (2285/2405)
remote: Compressing objects: 96% (2309/2405)
remote: Compressing objects: 97% (2333/2405)
remote: Compressing objects: 98% (2357/2405)
remote: Compressing objects: 99% (2381/2405)
remote: Compressing objects: 100% (2405/2405)
remote: Compressing objects: 100% (2405/2405), done.
Receiving objects: 0% (1/2506)
Receiving objects: 1% (26/2506)
Receiving objects: 2% (51/2506)
Receiving objects: 3% (76/2506)
Receiving objects: 4% (101/2506)
Receiving objects: 5% (126/2506)
Receiving objects: 6% (151/2506)
Receiving objects: 7% (176/2506)
Receiving objects: 8% (201/2506)
Receiving objects: 9% (226/2506)
Receiving objects: 10% (251/2506)
Receiving objects: 11% (276/2506)
Receiving objects: 12% (301/2506)
Receiving objects: 13% (326/2506)
Receiving objects: 14% (351/2506)
Receiving objects: 15% (376/2506)
Receiving objects: 16% (401/2506)
Receiving objects: 17% (427/2506)
Receiving objects: 18% (452/2506)
Receiving objects: 19% (477/2506)
Receiving objects: 20% (502/2506)
Receiving objects: 21% (527/2506)
Receiving objects: 22% (552/2506)
Receiving objects: 23% (577/2506)
Receiving objects: 24% (602/2506)
Receiving objects: 25% (627/2506)
Receiving objects: 26% (652/2506)
Receiving objects: 27% (677/2506)
Receiving objects: 28% (702/2506)
Receiving objects: 29% (727/2506)
Receiving objects: 30% (752/2506)
Receiving objects: 31% (777/2506)
Receiving objects: 32% (802/2506)
Receiving objects: 33% (827/2506)
Receiving objects: 34% (853/2506)
Receiving objects: 35% (878/2506)
Receiving objects: 36% (903/2506)
Receiving objects: 37% (928/2506)
Receiving objects: 38% (953/2506)
Receiving objects: 39% (978/2506)
Receiving objects: 40% (1003/2506)
Receiving objects: 41% (1028/2506)
Receiving objects: 42% (1053/2506)
Receiving objects: 43% (1078/2506)
Receiving objects: 44% (1103/2506)
Receiving objects: 45% (1128/2506)
Receiving objects: 46% (1153/2506)
Receiving objects: 47% (1178/2506)
Receiving objects: 48% (1203/2506)
Receiving objects: 49% (1228/2506)
Receiving objects: 50% (1253/2506)
Receiving objects: 51% (1279/2506)
Receiving objects: 52% (1304/2506)
Receiving objects: 53% (1329/2506)
Receiving objects: 54% (1354/2506)
Receiving objects: 55% (1379/2506)
Receiving objects: 56% (1404/2506)
Receiving objects: 57% (1429/2506)
Receiving objects: 58% (1454/2506)
Receiving objects: 59% (1479/2506)
Receiving objects: 60% (1504/2506)
Receiving objects: 61% (1529/2506)
Receiving objects: 62% (1554/2506)
Receiving objects: 63% (1579/2506)
Receiving objects: 64% (1604/2506)
Receiving objects: 65% (1629/2506)
Receiving objects: 66% (1654/2506)
Receiving objects: 67% (1680/2506)
Receiving objects: 68% (1705/2506)
Receiving objects: 69% (1730/2506)
Receiving objects: 70% (1755/2506)
Receiving objects: 71% (1780/2506)
Receiving objects: 72% (1805/2506)
Receiving objects: 73% (1830/2506)
Receiving objects: 74% (1855/2506)
Receiving objects: 75% (1880/2506), 188.00 KiB | 367.00 KiB/s
Receiving objects: 76% (1905/2506), 188.00 KiB | 367.00 KiB/s
Receiving objects: 77% (1930/2506), 188.00 KiB | 367.00 KiB/s
Receiving objects: 78% (1955/2506), 188.00 KiB | 367.00 KiB/s
Receiving objects: 78% (1958/2506), 412.00 KiB | 406.00 KiB/s
Receiving objects: 79% (1980/2506), 412.00 KiB | 406.00 KiB/s
Receiving objects: 80% (2005/2506), 652.00 KiB | 429.00 KiB/s
Receiving objects: 81% (2030/2506), 652.00 KiB | 429.00 KiB/s
Receiving objects: 82% (2055/2506), 652.00 KiB | 429.00 KiB/s
Receiving objects: 83% (2080/2506), 652.00 KiB | 429.00 KiB/s
Receiving objects: 84% (2106/2506), 652.00 KiB | 429.00 KiB/s
Receiving objects: 85% (2131/2506), 652.00 KiB | 429.00 KiB/s
Receiving objects: 86% (2156/2506), 652.00 KiB | 429.00 KiB/s
Receiving objects: 86% (2170/2506), 964.00 KiB | 468.00 KiB/s
Receiving objects: 87% (2181/2506), 964.00 KiB | 468.00 KiB/s
Receiving objects: 88% (2206/2506), 964.00 KiB | 468.00 KiB/s
Receiving objects: 89% (2231/2506), 964.00 KiB | 468.00 KiB/s
Receiving objects: 90% (2256/2506), 964.00 KiB | 468.00 KiB/s
Receiving objects: 90% (2266/2506), 1.32 MiB | 416.00 KiB/s
Receiving objects: 91% (2281/2506), 1.32 MiB | 416.00 KiB/s
Receiving objects: 92% (2306/2506), 1.32 MiB | 416.00 KiB/s
Receiving objects: 92% (2312/2506), 1.54 MiB | 411.00 KiB/s
Receiving objects: 92% (2319/2506), 1.71 MiB | 347.00 KiB/s
Receiving objects: 93% (2331/2506), 1.77 MiB | 306.00 KiB/s
Receiving objects: 93% (2333/2506), 1.77 MiB | 306.00 KiB/s
Receiving objects: 93% (2341/2506), 1.82 MiB | 222.00 KiB/s
Receiving objects: 93% (2344/2506), 1.88 MiB | 112.00 KiB/s
Receiving objects: 93% (2345/2506), 1.90 MiB | 97.00 KiB/s
Receiving objects: 93% (2348/2506), 1.93 MiB | 65.00 KiB/s
Receiving objects: 94% (2356/2506), 1.93 MiB | 65.00 KiB/s
Receiving objects: 94% (2371/2506), 1.98 MiB | 46.00 KiB/s
Receiving objects: 94% (2371/2506), 2.01 MiB | 34.00 KiB/s
Receiving objects: 94% (2372/2506), 2.01 MiB | 34.00 KiB/s
Receiving objects: 94% (2372/2506), 2.04 MiB | 29.00 KiB/s
Receiving objects: 94% (2373/2506), 2.05 MiB | 28.00 KiB/s
Receiving objects: 94% (2375/2506), 2.07 MiB | 22.00 KiB/s
Receiving objects: 94% (2376/2506), 2.09 MiB | 21.00 KiB/s
Receiving objects: 95% (2381/2506), 2.11 MiB | 19.00 KiB/s
error: index-pack died of signal 15
fatal: index-pack failed
Solution
Harish Talanki's suggestion helped to narrow down the issue. It was a DNS issue on the stash. The GIT TRACE identified the same.
Answered By - Jeel
