[FIXED] jenkins fails to start correctly, despite lack of error output

Issue

So, I've seen answers for when Jenkins totally fails to start. It looks like I have a slightly different problem where Jenkins runs but then immediately exits.

After an EC2 instance reboot, I tried this:

λ ip-xx-xxx-xx-xx jenkinshome → sudo service jenkins start
Starting jenkins (via systemctl):                          [  OK  ]

λ ip-xx-xxx-xx-xx jenkinshome → sudo service jenkins status
● jenkins.service - LSB: Jenkins Automation Server
Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
Active: active (exited) since Fri 2020-11-13 22:32:56 UTC; 3s ago
 Docs: man:systemd-sysv-generator(8)
 Process: 14875 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)
 Process: 14897 ExecStart=/etc/rc.d/init.d/jenkins start (code=exited, status=0/SUCCESS)

 Nov 13 22:32:55 xx...xx systemd[1]: Starting LSB: Jenkins Automation Server...
 Nov 13 22:32:55 xx...xx runuser[14902]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)
 Nov 13 22:32:56 xx...xx systemd[1]: Started LSB: Jenkins Automation Server.
 Nov 13 22:32:56 xx...xx jenkins[14897]: Starting Jenkins [  OK  ] 

Additionally, running java -jar /usr/lib/jenkins/jenkins.war just seems to install and start a new jenkins instance, rather than starting my previously running jenkins instance.

Worth noting: I'm at very little free space left on my device, and there are significant admin hurdles for cleaning disk space and for getting more space allocated, though this may be related.

Not quite sure how to proceed, but any tips are welcome. Thanks!

Solution

The answer was fairly straightforward, and dare I say obvious: I needed to reinstall jenkins. I'm on RHEL, but this could be adapted for ubuntu, etc. (see link below).

# stop the jenkins service altogether
sudo service jenkins stop

# backup jenkins_home files
sudo cp -r /var/lib/jenkins ~/.jenkins_home_backup 

# remove the jenkins installation entirely
sudo yum remove jenkins # remove existing jenkins install

# add the jenkins ssh key locally
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

# reinstall jenkins
sudo yum -y install jenkins

# start jenkins!
sudo service jenkins start

More or less adapted from: https://computingforgeeks.com/how-to-install-jenkins-on-centos-rhel-8/

You may need to also reference: https://www.jenkins.io/blog/2020/07/27/repository-signing-keys-changing/

Answered By - jkix