[FIXED] "Each" Closure and For Loop in Jenkins Parallel script issue

Issue

I have a Jenkins pipeline that basically reads a JSON file and prints out some values from it.

import groovy.json.JsonSlurper
import groovy.json.JsonSlurperClassic


pipeline {
    agent any
    stages {
        stage('test') {
            steps {
                script {
                    def environment = new JsonSlurperClassic().parseText('''
                        {                              
                          "list": [
                            {
                              "name": "service1"
                            },
                            {
                              "name": "service2"
                            },
                            {
                              "name": "NotAService"
                            },
                            {
                              "name": "AnotherDummyService-mock",
                            }  
                          ]
                        }
                        '''
                        )

                    def forLoopBuilders = [:]
                    for (artifact in environment.list) {
                        if (!artifact.name.contains("-mock")) {
                            println("Before parallel, the value is ${artifact.name}")
                            forLoopBuilders[artifact.name] = { println(artifact.name) }
                        }
                    }
                    parallel forLoopBuilders

                    def closureBuilders = [:]
                    environment.list.each { artifact ->
                        if (!artifact.name.contains("-mock")) {
                            println("Before parallel, the value is ${artifact.name}")
                            closureBuilders[artifact.name] = { println(artifact.name) }
                        }
                    }
                    parallel closureBuilders
                }
            }
        }
    }
}


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

The builders variable is how I store stages that will run in parallel. Basically it is [stageA: What to do in this stageA, anotherStage: What to do in this anotherStage]

The output is as below

10:20:53  Before parallel, the value is service1
10:20:53  [Pipeline] echo
10:20:53  Before parallel, the value is service2
10:20:53  [Pipeline] echo
10:20:53  Before parallel, the value is NotAService
10:20:53  [Pipeline] parallel
10:20:53  [Pipeline] { (Branch: service1)
10:20:53  [Pipeline] { (Branch: service2)
10:20:53  [Pipeline] { (Branch: NotAService)
10:20:53  [Pipeline] echo
10:20:53  AnotherDummyService-mock
10:20:53  [Pipeline] }
10:20:53  [Pipeline] echo
10:20:53  AnotherDummyService-mock
10:20:53  [Pipeline] }
10:20:53  [Pipeline] echo
10:20:53  AnotherDummyService-mock

10:20:53  [Pipeline] // parallel
10:20:53  [Pipeline] echo
10:20:53  Before parallel, the value is service1
10:20:53  [Pipeline] echo
10:20:54  Before parallel, the value is service2
10:20:54  [Pipeline] echo
10:20:54  Before parallel, the value is NotAService
10:20:55  [Pipeline] parallel
10:20:55  [Pipeline] { (Branch: service1)
10:20:55  [Pipeline] { (Branch: service2)
10:20:55  [Pipeline] { (Branch: NotAService)
10:20:55  [Pipeline] echo
10:20:55  service1
10:20:55  [Pipeline] }
10:20:55  [Pipeline] echo
10:20:55  service2
10:20:55  [Pipeline] }
10:20:55  [Pipeline] echo
10:20:55  NotAService

As you can see the outputs from running the parallel stages are different. Why is that so?

What I want is the output from parallel forLoopBuilders should be the same as that from closureBuilders.

Solution

This seems to be the result of how closures within for loops capture the loop variable, artifact in your case.

There's only a single loop variable that keeps being re-assigned... so the closures capture that single variable, but once the loop ends, that variable will have the last value assigned to it, hence all closures will only see that value later.

You can see how that behaves by running this simple example in pure Groovy:

    def map = [
            'Service1': 's1',
            'Service2': 's2'
    ]
    def closures = []
    for (entry in map.entrySet()) {
        closures << { println "ENTRY: $entry" }
    }
    closures*.call()

This will print:

ENTRY: Service2=s2
ENTRY: Service2=s2

I.e. the closures capture the last value of the entry variable.

Groovy closures are smarter in capturing values, they actually create new variables on each run, so if you replace the for-loop with a each { ... } construct, it works:

    def map = [
            'Service1': 's1',
            'Service2': 's2'
    ]
    def closures = []
    map.entrySet().each { entry ->
        closures << { println "ENTRY: $entry" }
    }
    closures*.call()

Prints:

ENTRY: Service1=s1
ENTRY: Service2=s2

In your case, just just use each { } and it should do what you want.

EDIT

If you insist in using the for loop, you should do like you would do in Java and assign the current value to a new variable.

The following code has the same result as the one using each:

    def map = [
            'Service1': 's1',
            'Service2': 's2'
    ]
    def closures = []
    for (entry in map.entrySet()) {
        def value = entry
        closures << { println "ENTRY: $value" }
    }
    closures*.call()

Answered By - Renato