[FIXED] How does Jenkins choose a pipeline agent between multiple labels?

Issue

I have a Jenkins pipeline that I'd like to run on either a params-specified agent or master. The pipeline code that implements this is:

pipeline {
  agent { label "${params.agent} || master" }
  ...
}

I've surmised, from the following posts, that the || operator needs to be inside the (double-)quotes:
Can I define multiple agent labels in a declarative Jenkins Pipeline?
https://serverfault.com/questions/1074089/how-to-apply-multiple-labels-to-jenkins-nodes
Jenkinsfile - agent matching multiple labels

When I run this job, it seems to always run on master.
When I switch the order of ${params.agent} and master in the agent statement, it seems to still always run on master.
If I remove " || master" from the agent statement, then the job runs on the params-specified agent.

Question: Is my observation that Jenkins "prefers" master a coincidence, or is there something wrong with the syntax that's making Jenkins default to master?
Is there some way to have Jenkins prefer not-master so that I can test validity of the agent statement?

Solution

So, when Jenkins encounters the line

  agent { label "${params.agent} || master" }

it will do exactly one of the following:

1. schedule your job on one of the nodes that match that label; or
2. get stuck until there's a node that matches that label, or until aborted.

With regards to option 1, there's no guarantee that it will do a round-robin, a random choice, or prefer some nodes but not the others, etc. In practice, when several nodes match, Jenkins will prefer the node that ran your pipeline in the past. This is a reasonable behavior — if there's a workspace already on that node, some operations (like git checkout) may happen faster, saving time.

With regards to option 2, that's also a reasonable behavior. We actually use that to schedule a job on a non-existing label, while manipulating the labels to produce one that would match.

So, there's nothing wrong with your syntax, and Jenkins is behaving as designed.

If you want to implement some custom rule — like "always try a different node", or "try to use master as little as possible" — you have to code that.

Example pipeline (note I haven't checked it):

import hudson.model.Hudson

properties([
    parameters([
        string(name: 'DEPLOY_ON', defaultValue: 'node_name',
                description: 'try to run on this node, or master'),
    ])
])

resulting_node_name = ''

pipeline {
    agent { node { label 'master' } }
    stages {
        stage ('Do on master') {
            steps {
                script {
                    resulting_node_name = params.DEPLOY_ON
                    // note: this gets node by name, but you can get by label if you wish
                    def slave = Jenkins.instance.getNode(resulting_node_name)
                    
                    if (slave == null) {
                        currentBuild.result = 'FAILURE'
                    }

                    def computer = slave.computer
                    if (computer == null || computer.getChannel() == null || slave.name != params.DEPLOY_ON) {
                        println "Something wrong with the slave object, setting master"
                        resulting_node_name = 'master'
                    }

                    printSlaveInfo(slave)

                    computer = null
                    slave = null
                }
            }
        }

        stage('Do on actual node') {
            agent { node { label resulting_node_name } }
            steps {
                script {
                    println "Running on ${env.NODE_NAME}"
                }
            }

        }
    }
}

@NonCPS
def printSlaveInfo(slave) {
    // some info that you can use to choose the least-busy, best-equipped, etc.
    println('====================')
    println('Name: ' + slave.name)
    println('getLabelString: ' + slave.getLabelString())
    println('getNumExectutors: ' + slave.getNumExecutors())
    println('getRemoteFS: ' + slave.getRemoteFS())
    println('getMode: ' + slave.getMode())
    println('getRootPath: ' + slave.getRootPath())
    println('getDescriptor: ' + slave.getDescriptor())
    println('getComputer: ' + slave.getComputer())
    def computer = slave.computer
    println('\tcomputer.isAcceptingTasks: ' + computer.isAcceptingTasks())
    println('\tcomputer.isLaunchSupported: ' + computer.isLaunchSupported())
    println('\tcomputer.getConnectTime: ' + computer.getConnectTime())
    println('\tcomputer.getDemandStartMilliseconds: ' + computer.getDemandStartMilliseconds())
    println('\tcomputer.isOffline: ' + computer.isOffline())
    println('\tcomputer.countBusy: ' + computer.countBusy())
    println('\tcomputer.getLog: ' + computer.getLog())
    println('\tcomputer.getBuilds: ' + computer.getBuilds())
    println('====================')
}

Answered By - MaratC
Answer Checked By - Candace Johnson (JavaFixing Volunteer)