Issue
I am trying to use Jenkins multibranch pipelines to build my multi-platform code. I saw declarative matrix as a good approach to achieve DRY.
Anyway, it seems that builds for each axis element are executed on the same checked out folder: this leads to a fail of the concurrent builds. If I build for a single platform everything works fine.
I am sure 100% the workspace is shared because, in the end, the workspace folder contains both rootfs_A.tar and rootfs_B.tar.
How can I rework this in such a way that ./build.sh works on a separate workspace?
This is the actual Jenkinsfile I am using.
pipeline {
agent any
stages {
stage('Build') {
matrix {
axes {
axis {
name 'PLATFORM'
values 'A', 'B'
}
}
stages {
stage('Prep') {
steps {
sh "cp /opt/${PLATFORM}/rootfs.tar rootfs_${PLATFORM}.tar"
}
}
stage('Build') {
steps {
echo "Do Build for ${PLATFORM}"
sh "./build.sh -p ${PLATFORM}
}
}
}
}
}
}
}
Solution
You can configure matrix pipeline using the customWorkspace agent option to run each dimension on a separate workspace. Consider the following example:
pipeline {
agent none
stages {
stage("Build") {
matrix {
agent {
node {
label ""
customWorkspace "workspace/${JOB_NAME}/PLATFORM/${PLATFORM}/"
}
}
axes {
axis {
name "PLATFORM"
values "A", "B"
}
}
stages {
stage("Prepare") {
steps {
sh "pwd"
sh "ls -lah"
}
}
}
}
}
}
}
The crucial part is the agent configuration inside the matrix block. Here I use the node agent with an empty label (which is the equivalent of agent any from your example) and I set a custom workspace inside the JENKINS_HOME relative path. This configuration runs every dimension inside the separate workspace (including cloning the repository to each custom workspace.)
Below you can find an exemplary output that illustrates this declarative pipeline execution flow.
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] parallel
[Pipeline] { (Branch: Matrix - PLATFORM = 'A')
[Pipeline] { (Branch: Matrix - PLATFORM = 'B')
[Pipeline] stage
[Pipeline] { (Matrix - PLATFORM = 'A')
[Pipeline] stage
[Pipeline] { (Matrix - PLATFORM = 'B')
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] node
Running on Jenkins in /home/wololock/.jenkins/workspace/jenkins-matrix-workspaces_master
[Pipeline] node
[Pipeline] {
[Pipeline] ws
Running in /home/wololock/.jenkins/workspace/jenkins-matrix-workspaces/master/PLATFORM/A
[Pipeline] {
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository file:///home/wololock/workspace/jenkins-matrix-workspaces
> git init /home/wololock/.jenkins/workspace/jenkins-matrix-workspaces/master/PLATFORM/A # timeout=10
Fetching upstream changes from file:///home/wololock/workspace/jenkins-matrix-workspaces
> git --version # timeout=10
> git --version # 'git version 2.26.3'
> git fetch --no-tags --force --progress -- file:///home/wololock/workspace/jenkins-matrix-workspaces +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url file:///home/wololock/workspace/jenkins-matrix-workspaces # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision c6083445649b0c73b70aefcb1bc662c30577bd00 (master)
> git config core.sparsecheckout # timeout=10
> git checkout -f c6083445649b0c73b70aefcb1bc662c30577bd00 # timeout=10
Commit message: "update"
> git rev-list --no-walk 8d67e2047ad7a0e65899f9855ae289eda90ea974 # timeout=10
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Prepare)
[Pipeline] sh
+ pwd
/home/wololock/.jenkins/workspace/jenkins-matrix-workspaces/master/PLATFORM/A
[Pipeline] sh
+ ls -lah
total 20K
drwxrwxr-x 3 wololock wololock 4.0K Apr 21 12:17 .
drwxrwxr-x 4 wololock wololock 4.0K Apr 21 12:17 ..
-rw-rw-r-- 1 wololock wololock 11 Apr 21 12:17 file
drwxrwxr-x 8 wololock wololock 4.0K Apr 21 12:17 .git
-rw-rw-r-- 1 wololock wololock 762 Apr 21 12:17 Jenkinsfile
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // ws
[Pipeline] }
Running on Jenkins in /home/wololock/.jenkins/workspace/jenkins-matrix-workspaces_master
[Pipeline] // node
[Pipeline] {
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] ws
Running in /home/wololock/.jenkins/workspace/jenkins-matrix-workspaces/master/PLATFORM/B
[Pipeline] {
[Pipeline] // stage
[Pipeline] }
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository file:///home/wololock/workspace/jenkins-matrix-workspaces
> git init /home/wololock/.jenkins/workspace/jenkins-matrix-workspaces/master/PLATFORM/B # timeout=10
Fetching upstream changes from file:///home/wololock/workspace/jenkins-matrix-workspaces
> git --version # timeout=10
> git --version # 'git version 2.26.3'
> git fetch --no-tags --force --progress -- file:///home/wololock/workspace/jenkins-matrix-workspaces +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url file:///home/wololock/workspace/jenkins-matrix-workspaces # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision c6083445649b0c73b70aefcb1bc662c30577bd00 (master)
> git config core.sparsecheckout # timeout=10
> git checkout -f c6083445649b0c73b70aefcb1bc662c30577bd00 # timeout=10
Commit message: "update"
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Prepare)
[Pipeline] sh
+ pwd
/home/wololock/.jenkins/workspace/jenkins-matrix-workspaces/master/PLATFORM/B
[Pipeline] sh
+ ls -lah
total 20K
drwxrwxr-x 3 wololock wololock 4.0K Apr 21 12:17 .
drwxrwxr-x 6 wololock wololock 4.0K Apr 21 12:17 ..
-rw-rw-r-- 1 wololock wololock 11 Apr 21 12:17 file
drwxrwxr-x 8 wololock wololock 4.0K Apr 21 12:17 .git
-rw-rw-r-- 1 wololock wololock 762 Apr 21 12:17 Jenkinsfile
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // ws
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
Answered By - Szymon Stepniak
