Issue
Ansible code works fine & is able to read line-by-line when the passed filenames has no white spaces.
The issue is when Jenkins passes the filenames in source_file parameter having whitespaces as below.
Below is how ansible is called:
sh "ansible-playbook -i /web/playbooks/filecopy/allmwhosts.hosts /web/playbooks/filecopy/copyfiles.yml -e source_host=$source_host -e '{ source_file: $source_file }'
[Pipeline] sh (hide)
+ ansible-playbook -i /web/filecopy/allmwhosts.hosts /web/playbooks/filecopy/copyfiles.yml -e source_host=usurb31as30v -e '{ source_file: /web/lib/hello.txt
/web/lib/Policy Form_LifeAgents_new.pdf
/web/lib/Policy Form_LifeAgents_old.pdf }'
Ansible code below:
- debug:
msg: "FILES: {{ files_list }}"
- debug:
msg: "The NEWLINE files are {{ item }}"
with_items:
- "{{ source_file.split() }}"
- debug:
msg: "The NEWLINE1 files are {{ item }}"
with_items:
- "{{ source_file.splitlines() }}"
- set_fact:
source_file_new: "{{ item | quote + '\n' + source_file | default() }}"
with_items:
- "{{ source_file.split('\n') }}"
- debug:
msg: "AFTER QUOTE: {{ source_file_new }}"
- debug:
msg: "QUOTED NEWLINE files are {{ item }}"
with_items:
- "{{ source_file_new.split('\n') }}"
Output:
TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:18
Saturday 20 August 2022 22:59:50 -0500 (0:00:00.077) 0:00:00.118 *******
ok: [localhost] => {
"msg": "FILES: /web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}
TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:21
Saturday 20 August 2022 22:59:50 -0500 (0:00:00.032) 0:00:00.150 *******
ok: [localhost] => (item=/web/lib/hello.txt) => {
"msg": "The NEWLINE files are /web/lib/hello.txt"
}
ok: [localhost] => (item=/web/lib/Policy) => {
"msg": "The NEWLINE files are /web/lib/Policy"
}
ok: [localhost] => (item=Form_LifeAgents_new.pdf) => {
"msg": "The NEWLINE files are Form_LifeAgents_new.pdf"
}
ok: [localhost] => (item=/web/lib/Policy) => {
"msg": "The NEWLINE files are /web/lib/Policy"
}
ok: [localhost] => (item=Form_LifeAgents_old.pdf) => {
"msg": "The NEWLINE files are Form_LifeAgents_old.pdf"
}
TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:21
Sunday 21 August 2022 12:16:24 -0500 (0:00:00.021) 0:00:00.101 *********
ok: [localhost] => (item=/web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf /web/lib/Policy Specimen_MLA_Everest_new.pdf /web/lib/Policy Specimen_MLA_Everest_old.pdf) => {
"msg": "The NEWLINE1 files are /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf /web/lib/Policy Specimen_MLA_Everest_new.pdf /web/lib/Policy Specimen_MLA_Everest_old.pdf"
}
TASK [set_fact] ****************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:26
Saturday 20 August 2022 22:59:50 -0500 (0:00:00.069) 0:00:00.219 *******
ok: [localhost] => (item=/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf) => {
"ansible_facts": {
"source_file_new": "'/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf'\n/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
},
"ansible_loop_var": "item",
"changed": false,
"item": "/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}
TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:31
Saturday 20 August 2022 22:59:50 -0500 (0:00:00.080) 0:00:00.300 *******
ok: [localhost] => {
"msg": "AFTER QUOTE: '/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf'\n/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}
TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:35
Saturday 20 August 2022 22:59:50 -0500 (0:00:00.031) 0:00:00.331 *******
ok: [localhost] => (item='/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf') => {
"msg": "QUOTED NEWLINE files are '/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf'"
}
ok: [localhost] => (item=/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf) => {
"msg": "QUOTED NEWLINE files are /web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}
The expectation was three lines should have been read by ansible however neither the split() or split('\n') is helping
Solution
You are launching your Ansible playbook through shell. The problem here is a quoting issue. (ba)sh interprets new lines and spaces as parameter separators. So you need to quote everything correctly in order for:
- (ba)sh to understand the
-eparameter to Ansible ends after the last file in the list - Ansible to understand that this is a single string containing new lines and spaces.
Once this is fixed the rest of the processing is trivial since you just have to source_file.splitlines() your value to get your list of individual files.
Note that Jenkins has an Ansible plugin that can ease your life in such cases on the gui. If you add a "Invoke ansible playbook" build step and click on "advanced", you will be able to add extra variables by simply entering the key name and the value (which can be a variable). You can see the correct secure quoting in the result when launching your job.
Unfortunately, this handy help is not available in the pipeline syntax helper where you will also have to construct your string of extra parameters by yourself.
Long story short: this is how your shell script command should be written to securely quote all your ansible parameters:
ansible-playbook -i /web/playbooks/filecopy/allmwhosts.hosts \
/web/playbooks/filecopy/copyfiles.yml \
-e "source_host='$source_host'" \
-e "source_file='$source_file'"
As a proof of concept, here is an all-in-one demo declarative Jenkins pipeline that you can paste in a pipeline job and which successfully runs the in-declared playbook both through shell and the Ansible plugin.
Note that since I declared the job parameters inside the pipeline, the first build will systematically fail. You will only be able to provide the params interactively on the second build. The defaults should be enough reproduce the situation described in your question.
pipeline {
agent any
stages {
stage('Setup parameters') {
steps {
script {
properties([
parameters([
text(
defaultValue: "/a/file\n/an/other file\n/one/more/file",
description: "Enter absolute path of files on target one on each line",
name: "source_file"
)
])
])
}
}
}
stage('Create playbook') {
steps {
sh '''
cat <<"EOF" > playbook.yml
---
- hosts: localhost
gather_facts: false
vars:
file_list: "{{ source_file.splitlines() }}"
tasks:
- name: Show content of processed incoming extra var
debug:
var: file_list
- name: Just proove it works
debug:
msg: "received individual file {{ item }}"
loop: "{{ file_list }}"
EOF
'''
}
}
stage('Run playbook through shell'){
steps {
sh '''
ansible-playbook playbook.yml -e "source_file='$source_file'"
'''
}
}
stage('Run playbook through ansible plugin'){
steps {
ansiblePlaybook extras: '-e "source_file=\'$source_file\'"', playbook: 'playbook.yml'
}
}
}
}
Which gives (on second run and using the default values):
Started by user Olivier Clavel
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/testpipe
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Setup parameters)
[Pipeline] script
[Pipeline] {
[Pipeline] properties
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Create playbook)
[Pipeline] sh
+ cat
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run playbook through shell)
[Pipeline] sh
+ ansible-playbook playbook.yml -e source_file='/a/file
/an/other file
/one/more/file'
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [Show content of processed incoming extra var] ****************************
ok: [localhost] => {
"file_list": [
"/a/file",
"/an/other file",
"/one/more/file"
]
}
TASK [Just proove it works] ****************************************************
ok: [localhost] => (item=/a/file) => {
"msg": "received individual file /a/file"
}
ok: [localhost] => (item=/an/other file) => {
"msg": "received individual file /an/other file"
}
ok: [localhost] => (item=/one/more/file) => {
"msg": "received individual file /one/more/file"
}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run playbook through ansible plugin)
[Pipeline] ansiblePlaybook
[testpipe] $ ansible-playbook playbook.yml -e "source_file='/a/file
/an/other file
/one/more/file'"
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [Show content of processed incoming extra var] ****************************
ok: [localhost] => {
"file_list": [
"/a/file",
"/an/other file",
"/one/more/file"
]
}
TASK [Just proove it works] ****************************************************
ok: [localhost] => (item=/a/file) => {
"msg": "received individual file /a/file"
}
ok: [localhost] => (item=/an/other file) => {
"msg": "received individual file /an/other file"
}
ok: [localhost] => (item=/one/more/file) => {
"msg": "received individual file /one/more/file"
}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Answered By - Zeitounator
Answer Checked By - David Marino (JavaFixing Volunteer)
