[FIXED] Multiple job for same cronjob with different params in Jenkins

Issue

We are using a third-party service to create and use vouchers. There are 80k+ vouchers already made. One of our cronjobs checks the status (used/unused) of each voucher one by one synchronously and updates it in our server database. It takes 2hours to complete one pass, then it continues from the first voucher for the next pass.

Constraints:

• the third party supports the 6 queries per second(QPS).
• We have only a primary Jenkins server and no agent nodes.

With one Jenkins server can we improve the execution time?

Can we set up multiple jobs executing parallelly on a primary Jenkins server for the same cronjob? Like the first 50k records are processed by one of the jobs and the rest are processed by another.

Solution

If you have room to vertical scale your VM in case you hit a resource(CPU, Memory) bottleneck, you should be able to achieve the performance. IMV best option is using Parallel stages in your Pipeline. If you know the batch sizes beforehand, you can hardcode the sizes within each stage, If you want to add some logic to determine how many records you have and then based on that allocate records, you can create a Pipeline with dynamic stages, something like below.

pipeline {
    agent any

    stages {
        stage('Parallel') {
            steps {
                script {
                    parallel parallelJobs()
                }
            }
        }
    }
}


def getBatches() {
  // Have some logic to allocate batches
  return ["1-20000", "20000-30000", "30000-50000"]
}

def parallelJobs() {
  jobs = [:]

  for (batch in getBatches()) {
    jobs[batch] = { stage(batch) {
       echo "Processing Batch $batch"
     }
    }
  }
  return jobs
}

Answered By - ycr
Answer Checked By - Clifford M. (JavaFixing Volunteer)