[FIXED] Can I make skipped tests more prominent in the Jenkins dashboard?

Issue

I have a project under test in Jenkins ("test installation"; it does some regression tests ti verify that the installer works). This project has a soft dependency on stuff outside of Jenkins' control: if the latest installer isn't available, then we can't test it. We can always re-test an old installer, though, and that seems worth doing (we've got the CPU cycles, so we may as well burn 'em).

What I'd like is produce a loud warning if the installer isn't current, but then continue with the tests.

The first thing I tried was making a test that failed when the installer was out of date. That was prominent, but confusing because the installation tests weren't actually the thing that was failing.

Now I have the same test, but it uses a JUnit assumption instead of an assertion, which means that the test either skips or passes. This is also less-than-perfect, because Jenkins reports "9 tests, 0 failures" on the front page, and it's only when I drill down multiple layers into the test results that I see that 1 of the 9 tests was skipped.

Can I get Jenkins to report skipped tests on the front page? I didn't find an appropriate-looking plug-in for it. Is there a better method I should use to warn about the installer being out of date?

Solution

A bit late to answer question for you but might be of help to other people . . .

Two easy enough things that I think would work best for cases like this:

• Add one extra test to check latest installer version and fail. So that job would be marked as unstable.

• Or you could use one of the post-build plugins to check logs and mark job as failed instead of unstable.

There is not an easy way to make skipped tests more prominent.

But you could do some post-processing on test results.

We generate a VERSION.txt file in job script/test scripts and put it in job workspace. Then we use a Groovy Postbuild action and set the job description:

"Groovy postbuild"

def currentBuild = Thread.currentThread().executable
def ws = manager.build.workspace.getRemote()
String desc = new File(ws + "/VERSION.txt").text
currentBuild.setDescription(desc)

This is quite useful we can see the version tested or other details in jobs history.

To mark things more prominently . . . >;) you could use a badges and some groovy.

Plugins used:

https://wiki.jenkins.io/display/JENKINS/Groovy+Postbuild+Plugin

The Groovy Postbuild plugin is the only one really needed. THe Badges API is part of Groovy Postbuild plugin.

https://wiki.jenkins.io/display/JENKINS/Groovy+plugin

The Groovy plugin is useful for experimenting with groovy or making jobs with groovy.

https://wiki.jenkins.io/display/JENKINS/Build+Trigger+Badge+Plugin

Actually the badges are available in jenkins groovy postbuild plugin. The BuildTriggerBadge plugin which I use is useful to have anyway if a variety of triggers are used but not actually needed for setting badge. I include it here as it is installed and I am not 100% sure my code would work without it (but I am maybe 98.5% sure). I do not have the Badge plugin installed.

See below some groovy experimentation with badges:

def currentBuild = Thread.currentThread().executable
def ws = manager.build.workspace.getRemote()
String desc = new File(ws + "/VERSION.txt").text
currentBuild.setDescription(desc)

if (desc.contains("ERROR")) {
  coverageText="VarkeninK, SomethinK iz bad."
  // Apologies :-7 I can only assume this was from Viktor in http://www.userfriendly.org/ web comic
  manager.addShortText(coverageText, "black", "repeating-linear-gradient(45deg, 
yellow, yellow 10px, Orange 10px, Orange 20px)", "0px", "white")
}

manager.addShortText("GreyWhite0pxWhite", "grey", "white", "0px", "white")
manager.addShortText("BlackGreen0pxWhite", "black", "green", "0px", "white")
manager.addShortText("BlackGreen5pxWhite", "black", "green", "5px", "white")
manager.addShortText("VERSION WhiteGreen0pxWhite", "white", "green", "0px", "white")
manager.addShortText("WhiteGreen5pxWhite", "white", "green", "5px", "white")

manager.addShortText("VERSION Black on Lime Green", "black", "limegreen", "0px", "white")

// darkgrey is lighter than grey!! :-P
manager.addShortText("OBSOLETE YellowDarkGrey5pxGrey", "yellow", "darkgrey", "5px", "grey")
manager.addShortText("OBSOLETE YellowGrey5pxGrey", "yellow", "grey", "5px", "grey")


manager.removeBadges()


manager.addShortText("VERSION Black on Lime Green", "black", "limegreen", "0px", "white")
manager.addShortText(desc, "black", "limegreen", "5px", "white")
manager.addShortText("OBSOLETE YellowGrey5pxGrey", "yellow", "grey", "5px", "grey")

manager.addBadge("warning.gif", "Warning test")
manager.addWarningBadge("other warning test")


// https://wiki.jenkins.io/display/JENKINS/Groovy+Postbuild+Plugin

// contains(file, regexp) - returns true if the given file contains a line matching regexp.
// logContains(regexp) - returns true if the build log file contains a line matching regexp.
// getMatcher(file, regexp) - returns a java.util.regex.Matcher for the first occurrence of regexp in the given file.
// getLogMatcher(regexp) - returns a java.util.regex.Matcher for the first occurrence of regexp in the build log file.
// setBuildNumber(number) - sets the build with the given number as current build. The current build is the target of all methods that add or remove badges and summaries or change the build result.
// addShortText(text) - puts a badge with a short text, using the default format.
// addShortText(text, color, background, border, borderColor) - puts a badge with a short text, using the specified format.
// addBadge(icon, text) - puts a badge with the given icon and text. In addition to the 16x16 icons offered by Jenkins, groovy-postbuild provides the following icons:
//  - completed.gif
//  - db_in.gif
//  - db_out.gif
//  - delete.gif
//  - error.gif
//  - folder.gif
//  - green.gif
//  - info.gif
//  - red.gif
//  - save.gif
//  - success.gif
//  - text.gif
//  - warning.gif
//  - yellow.gif
// addBadge(icon, text, link) - like addBadge(icon, text), but the Badge icon then actually links to the given link (since 1.8)
// addInfoBadge(text) - puts a badge with  info icon and the given text.
// addWarningBadge(text) - puts a badge with  warning icon and the given text.
// addErrorBadge(text) - puts a badge with  error icon and the given text.
// removeBadges() - removes all badges from the current build.
// removeBadge(index) - removes the badge with the given index.
// createSummary(icon) - creates an entry in the build summary page and returns a summary object corresponding to this entry. The icon must be one of the 48x48 icons offered by Jenkins. You can append text to the summary object by calling its appendText methods:
// appendText(text, escapeHtml)
// appendText(text, escapeHtml, bold, italic, color)
// removeSummaries() - removes all summaries from the current build.
// removeSummary(index) - removes the summary with the given index.
// buildUnstable() - sets the build result to UNSTABLE.
// buildFailure() - sets the build result to FAILURE.
// buildSuccess() - sets the build result to SUCCESS.

Answered By - gaoithe
Answer Checked By - Dawn Plyler (JavaFixing Volunteer)