[FIXED] How to run dynamic stages in paralell on jenkins with a separate kubernetes agent for each stage

Issue

I tried combining things I have found on the syntax but this is as close as I can get. It creates multiple stages but says they have no steps.

I can get it to run a bunch of parallel steps on the same agent if I move the agent syntax down to where the "test" stage is defined but I want to spin up separate pods for each one so I can actually use the kubernetes cluster effectively and do my work parallel.

attached is an example Jenkinsfile for reference

def parallelStagesMap

def generateStage(job) {
    return {
        stage ("$job.key") {
            agent {
                kubernetes {
                    cloud 'kubernetes'
                    yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: name
    image: image
    command:
    - sleep
    args:
    - infinity
"""
                }
            }
            steps {
                sh """
                    do some important stuff
                """
            }
        }
    }
}

pipeline {
    agent none
    stages {
        stage('Create List of Stages to run in Parallel') {
            steps {
                script {
                    def map = [
                        "name" : "aparam",
                        "name2" : "aparam2"
                    ]
                    parallelStagesMap = map.collectEntries {
                        ["${it.key}" : generateStage(it)]
                    }
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    parallel parallelStagesMap 
                }
            }
        }
        stage('Release') {
            agent etc
            steps {
                etc
            }
        }
    }
}

Solution

To run your dynamically created jobs in parallel you will have to use scripted pipeline syntax.
The equivalent syntax for the declarative kubernetes agent in the scripted pipeline is podTemplate and node (see full Doucumentation):

podTemplate(yaml: '''
    apiVersion: v1
    kind: Pod
    spec:
      containers:
      - name: maven
        image: maven:3.8.1-jdk-8
        command:
        - sleep
        args:
        - 99d
''') {
  node(POD_LABEL) {
     ...
    }
}

Notice that the podTemplate can receive the cloud parameter in addition to the yaml but it defaults to kubernetes so there is no need to pass it.

So in your case you can use this syntax to run the jobs in parallel on different agents:

// Assuming yaml is same for all nodes - if not it can be passed as parameter
podYaml= """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: name
    image: image
    command:
    - sleep
    args:
    - infinity
"""

pipeline {
    agent none
    stages {
        stage('Create List of Stages to run in Parallel') {
            steps {
                script {
                    def map = ["name" : "aparam",
                              "name2" : "aparam2"]
                    parallel map.collectEntries {
                        ["${it.key}" : generateStage(it)]
                    }
                }
            }
        }
    }
}

def generateStage(job) {
    return {
        stage(job.key) {
            podTemplate(yaml:podYaml)  {
                node(POD_LABEL) {
                    // Each execution runs on its own node (pod)
                    sh "do some important stuff with ${job.value}"
                }
            }
        }
    }
}

Answered By - Noam Helmer
Answer Checked By - Clifford M. (JavaFixing Volunteer)