Issue
I am writing a Jenkins job that will move files between two chrooted directories on a remote server.
This uses a Jenkins multiline string variable to store one or more file name, one per line.
The following will work for files without special characters or spaces:
## Jenkins parameters
# accountAlias = "test"
# sftpDir = "/path/to/chrooted home"
# srcDir = "/path/to/get/files"
# destDir = "/path/to/put/files"
# fileName = "file names # multiline Jenkins shell parameter, one file name per
#!/bin/bash
ssh user@server << EOF
#!/bin/bash
printf "\nCopying following file(s) from "${accountAlias}"_old account to "${accountAlias}"_new account:\n"
# Exit if no filename is given so Rsync does not copy all files in src directory.
if [ -z "${fileName}" ]; then
printf "\n***** At least one filename is required! *****\n"
exit 1
else
# While reading each line of fileName
while IFS= read -r line; do
printf "\n/"${sftpDir}"/"${accountAlias}"_old/"${srcDir}"/"\${line}" -> /"${sftpDir}"/"${accountAlias}"_new/"${destDir}"/"\${line}"\n"
# Rsync the files from old account to new account
# -v | verbose
# -c | replace existing files based on checksum, not timestamp or size
# -r | recursively copy
# -t | preserve timestamps
# -h | human readable file sizes
# -P | resume incomplete files + show progress bars for large files
# -s | Sends file names without interpreting special chars
sudo rsync -vcrthPs /"${sftpDir}"/"${accountAlias}"_old/"${srcDir}"/"\${line}" /"${sftpDir}"/"${accountAlias}"_new/"${destDir}"/"\${line}"
done <<< "${fileName}"
fi
printf "\nEnsuring all new files are owned by the "${accountAlias}"_new account:\n"
sudo chown -vR "${accountAlias}"_new:"${accountAlias}"_new /"${sftpDir}"/"${accountAlias}"_new/"${destDir}"
EOF
Using the file name "sudo bash -c 'echo "hello" > f.txt'.txt" as a test, my script will fail after the "sudo" in the file name.
I believe my problem that my $line variable are not properly quoted or escaped, resulting in bash not treating the $line value as one string.
I have tried single quotes or using awk/sed to insert back slashes in variable string, but this hasn't worked.
My theory is I am running into a problem with special chars and heredocs.
Solution
Although it's unclear to me from your description exactly what error you are encountering or where, you do have several problems in the script presented.
The main one might simply be the sudo command that you're trying to execute on the remote side. Unless user has passwordless sudo privilege (rather dangerous) sudo will prompt for a password and attempt to read it from the user's terminal. You are not providing a password. You could probably just interpolate it into the command stream (in the here doc) if in fact you collect it. Nevertheless, there is still a potential problem with that, as you perform potentially many sudo commands, and they may or may not request passwords depending on remote sudo configuration and the time between sudo commands. Best would be to structure the command stream so that only one sudo execution is required.
Additional considerations follow.
## Jenkins parameters # accountAlias = "test" # sftpDir = "/path/to/chrooted home" # srcDir = "/path/to/get/files" # destDir = "/path/to/put/files" # fileName = "file names # multiline Jenkins shell parameter, one file name per #!/bin/bash
The #!/bin/bash there is not the first line of the script, so it does not function as a shebang line. Instead, it is just an ordinary comment. As a result, when the script is executed directly, it might or might not be bash that runs it, and if if it is bash, it might or might not be running in POSIX compatibility mode.
ssh user@server << EOF #!/bin/bash
This #!/bin/bash is not a shebang line either, because that applies only to scripts read from regular files. As a result, the following commands are run by user's default shell, whatever that happens to be. If you want to ensure that the rest is run by bash, then perhaps you should execute bash explicitly.
printf "\nCopying following file(s) from "${accountAlias}"_old account to "${accountAlias}"_new account:\n"
The two expansions of $accountAlias (by the local shell) result in unquoted text passed to printf in the remote shell. You could consider just removing the de-quoting, but that would still leave you susceptible to malicious accountAlias values that included double-quote characters. Remember that these will be expanded on the local side, before the command is sent over the wire, and then the data will be processed by a remote shell, which is the one that will interpret the quoting.
This can be resolved by
Outside the heredoc, preparing a version of the account alias that can be safely presented to the remote shell
accountAlias_safe=$(printf %q "$accountAlias")and
Inside the heredoc, expanding it unquoted. I would furthermore suggest passing it as a separate argument instead of interpolating it into the larger string.
printf "\nCopying following file(s) from %s_old account to %s_new account:\n" ${accountAlias_safe} ${accountAlias_safe}
Similar applies to most of the other places where variables from the local shell are interpolated into the heredoc.
Here ...
# Exit if no filename is given so Rsync does not copy all files in src directory. if [ -z "${fileName}" ]; then
... why are you performing this test on the remote side? You would save yourself some trouble by performing it on the local side instead.
Here ...
printf "\n/"${sftpDir}"/"${accountAlias}"_old/"${srcDir}"/"\${line}" -> /"${sftpDir}"/"${accountAlias}"_new/"${destDir}"/"\${line}"\n"
... remote shell variable $line is used unquoted in the printf command. Its appearance should be quoted. Also, since you use the source and destination names twice each, it would be cleaner and clearer to put them in (remote-side) variables. AND, if the directory names have the form presented in comments in the script, then you are introducing excess / characters (though these probably are not harmful).
Good for you, documenting the meaning of all the rsync options used, but why are you sending all that over the wire to the remote side?
Also, you probably want to include rsync's -p option to preserve the same permissions. Possibly you want to include the -l option too, to copy any symbolic link as symbolic links.
Putting all that together, something more like this (untested) is probably in order:
#!/bin/bash
## Jenkins parameters
# accountAlias = "test"
# sftpDir = "/path/to/chrooted home"
# srcDir = "/path/to/get/files"
# destDir = "/path/to/put/files"
# fileName = "file names # multiline Jenkins shell parameter, one file name per
# Exit if no filename is given so Rsync does not copy all files in src directory.
if [ -z "${fileName}" ]; then
printf "\n***** At least one filename is required! *****\n"
exit 1
fi
accountAlias_safe=$(printf %q "$accountAlias")
sftpDir_safe=$(printf %q "$sftpDir")
srcDir_safe=$(printf %q "$srcDir")
destDir_safe=$(printf %q "$destDir")
fileName_safe=$(printf %q "$fileName")
IFS= read -r -p 'password for user@server: ' -s -t 60 password || {
echo 'password not entered in time' 1>&2
exit 1
}
# Rsync options used:
# -v | verbose
# -c | replace existing files based on checksum, not timestamp or size
# -r | recursively copy
# -t | preserve timestamps
# -h | human readable file sizes
# -P | resume incomplete files + show progress bars for large files
# -s | Sends file names without interpreting special chars
# -p | preserve file permissions
# -l | copy symbolic links as links
ssh user@server /bin/bash << EOF
printf "\nCopying following file(s) from %s_old account to %s_new account:\n" ${accountAlias_safe} ${accountAlias_safe}
sudo /bin/bash -c '
while IFS= read -r line; do
src=${sftpDir_safe}${accountAlias_safe}_old${srcDir_safe}"/\${line}"
dest="${sftpDir_safe}/${accountAlias_safe}_new${destDir_safe}/"\${line}"
printf "\n\${src} -> \${dest}\n"
rsync -vcrthPspl "\${src}" "\${dest}"
done <<<'${fileName_safe}'
printf "\nEnsuring all new files are owned by the %s_new account:\n" ${accountAlias_safe}
chown -vR ${accountAlias_safe}_new:${accountAlias_safe}_new ${sftpDir_safe}/${accountAlias_safe}_new${destDir_safe}
'
${password}
EOF
Answered By - John Bollinger
Answer Checked By - David Marino (JavaFixing Volunteer)
