Wiki

User Tools

Site Tools

Trace: • start
bash:wordsplitting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
bash:wordsplitting [2021/01/09 15:15] – peterbash:wordsplitting [2021/01/09 15:51] (current) – [Controlling Word Splitting] peter
Line 4: Line 4:
  
 Understanding how your original command will be transformed by the shell is of paramount importance in writing robust scripts. Understanding how your original command will be transformed by the shell is of paramount importance in writing robust scripts.
 +
 +<WRAP info>
 +**NOTE:**  Shell commands execute some program with a specific set of arguments (as well as setting up environment variables, opening file descriptors, etc.).
 +
 +  * **Word splitting** is part of the process that determines what each of those arguments will be.
 +    * After **Word splitting** and **pathname expansion**, every resulting word becomes an argument to the program that the shell executes.
 +
 +Word splitting is performed on the results of almost all unquoted expansions.
 +
 +  * The result of the expansion is broken into separate words based on the characters of the IFS variable. 
 +
 +</WRAP>
  
 ---- ----
Line 10: Line 22:
  
 <code> <code>
-The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion. +The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion  
 +and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion. 
  
 For additional information on word splitting and argument handling in Bash, consider reading Arguments. For additional information on word splitting and argument handling in Bash, consider reading Arguments.
Line 16: Line 29:
  
 ---- ----
- 
  
 ===== What is Word Splitting? ===== ===== What is Word Splitting? =====
Line 22: Line 34:
 Let's write a little helper script that will show us the arguments as passed by the shell: Let's write a little helper script that will show us the arguments as passed by the shell:
  
-<code bash>+<file bash test.sh>
 #!/bin/sh - #!/bin/sh -
 printf "%d args:" "$#" printf "%d args:" "$#"
 printf " <%s>" "$@" printf " <%s>" "$@"
 echo echo
 +</file>
 +
 +----
 +
 +==== Make it executable ====
 +
 +<code bash>
 +chmod +x test.sh
 </code> </code>
  
-If you create a file named args with the above contents, make it executable with chmod +x args, and put it in one of the directories listed in echo "$PATH", then you could run the following command with the following output:+---- 
 + 
 +==== Run it ====
  
 <code bash> <code bash>
-args hello world "how are you?"+./test.sh hello world "how are you?"
 </code> </code>
  
Line 41: Line 63:
 </code> </code>
  
-The ultimate result of most shell commands is to execute some program with a specific set of arguments (as well as setting up environment variables, opening file descriptors, etc.).+<WRAP info> 
 +**NOTE:**  The helper program above receives the argument list as constructed by the shell, and shows it to us.
  
-Word splitting is part of the process that determines what each of those arguments will be -- after word splitting and pathname expansion, every resulting word becomes an argument to the program that the shell executes.+</WRAP>
  
-Our helper program above receives the argument list as constructed by the shell, and shows it to us.+----
  
-Word splitting is performed on the results of almost all unquoted expansions.  The result of the expansion is broken into separate words based on the characters of the IFS variable.  If IFS is not set, then it will be performed as if IFS contained a space, a tab, and a newline.+===== Test when IFS is not SET ===== 
 + 
 +If IFS is not set, then it will be performed as if IFS contained a space, a tab, and a newline.
  
 For example: For example:
Line 53: Line 78:
 <code bash> <code bash>
 var="This is a variable" var="This is a variable"
-args $var+test.sh $var 
 +</code> 
 + 
 +returns: 
 + 
 +<code bash>
 4 args: <This> <is> <a> <variable> 4 args: <This> <is> <a> <variable>
 </code> </code>
  
 ---- ----
 +
 +===== Test when IFS is SET =====
  
 An example using IFS: An example using IFS:
Line 63: Line 95:
 <code bash> <code bash>
 log=/var/log/qmail/current IFS=/ log=/var/log/qmail/current IFS=/
-args $log+test.sh $log 
 +</code> 
 + 
 +returns: 
 + 
 +<code bash>
 args: <> <var> <log> <qmail> <current> args: <> <var> <log> <qmail> <current>
 unset IFS unset IFS
Line 70: Line 107:
 ---- ----
  
-An example with CommandSubstitution:+===== Test with Command Substitution =====
  
 <code bash> <code bash>
 ls -l ls -l
 +</code>
 +
 +returns:
 +
 +<code bash>
 total 2864 total 2864
 -rw-r--r-- 1 greg greg 2919154 2001-05-23 00:48 Yello - Oh Yeah.mp3 -rw-r--r-- 1 greg greg 2919154 2001-05-23 00:48 Yello - Oh Yeah.mp3
 +</code>
  
-args $(ls -l)+Now run against the test script: 
 + 
 +<code bash> 
 +test.sh $(ls -l) 
 +</code> 
 + 
 +returns: 
 + 
 +<code bash>
 11 args: <-rw-r--r--> <1> <greg> <greg> <2919154> <2001-05-23> <00:48> <Yello> <-> <Oh> <Yeah.mp3> 11 args: <-rw-r--r--> <1> <greg> <greg> <2919154> <2001-05-23> <00:48> <Yello> <-> <Oh> <Yeah.mp3>
 </code> </code>
Line 85: Line 136:
 ===== Controlling Word Splitting ===== ===== Controlling Word Splitting =====
  
-As you can see above, we usually do not want to let word splitting occur when filenames are involved. (See BashPitfalls for a discussion of this particular issue.)+As you can see above, we usually do not want to let word splitting occur when filenames are involved.
  
-Double quoting an expansion suppresses word splitting, except in the special cases of "$@" and "${array[@]}":+Double quoting an expansion suppresses word splitting, except in the special cases of <nowiki>"$@"</nowiki> and <nowiki>"${array[@]}"</nowiki>: 
 + 
 +<code bash> 
 +var="This is a variable"; test.sh "$var" 
 +</code> 
 + 
 +returns:
  
 <code bash> <code bash>
-var="This is a variable"; args "$var" 
 1 args: <This is a variable> 1 args: <This is a variable>
 +</code>
  
-array=(testing, testing, "1 2 3"); args "${array[@]}"+---- 
 + 
 +<code bash> 
 +array=(testing, testing, "1 2 3"); test.sh "${array[@]}" 
 +</code> 
 + 
 +returns: 
 + 
 +<code bash>
 3 args: <testing,> <testing,> <1 2 3> 3 args: <testing,> <testing,> <1 2 3>
 </code> </code>
  
-"$@" causes each positional parameter to be expanded to a separate word; its array equivalent likewise causes each element of the array to be expanded to a separate word.+<WRAP info> 
 +**NOTE:**  **<nowiki>"$@"</nowiki>** causes each positional parameter to be expanded to a separate word; its array equivalent likewise causes each element of the array to be expanded to a separate word.
  
 There are very complicated rules involving whitespace characters in IFS. Quoting the man page again: There are very complicated rules involving whitespace characters in IFS. Quoting the man page again:
Line 108: Line 174:
  
 We won't explore those rules in depth here, except to note the part about sequences of non-whitespace characters. We won't explore those rules in depth here, except to note the part about sequences of non-whitespace characters.
 +
 +</WRAP>
 +
 +----
 +
 +===== IFS with non-whitespace characters =====
  
 If IFS contains non-whitespace characters, then empty words can be generated: If IFS contains non-whitespace characters, then empty words can be generated:
Line 113: Line 185:
 <code bash> <code bash>
 getent passwd sshd getent passwd sshd
 +</code>
 +
 +returns:
 +
 +<code bash>
 sshd:x:100:65534::/var/run/sshd:/usr/sbin/nologin sshd:x:100:65534::/var/run/sshd:/usr/sbin/nologin
 +</code>
  
-IFS=:; args $(getent passwd sshd)+---- 
 + 
 +Set IFS 
 + 
 +<code bash> 
 +IFS=:; test.sh $(getent passwd sshd) 
 +</code> 
 + 
 +returns: 
 + 
 +<code bash>
 7 args: <sshd> <x> <100> <65534> <> </var/run/sshd> </usr/sbin/nologin> 7 args: <sshd> <x> <100> <65534> <> </var/run/sshd> </usr/sbin/nologin>
 +</code>
  
 +Unset IFS
 +
 +<code bash>
 unset IFS unset IFS
 </code> </code>
  
-There was another empty word generated in one of our previous examples, where IFS was set to /.  The observant reader will have noticed, therefore, that non-whitespace IFS characters are not ignored at the beginning and end of expansions, the way whitespace IFS characters are.+<WRAP info> 
 +**NOTE:**  There was another empty word generated in one of our previous examples, where IFS was set to **/**.
  
-Whitespace IFS characters get consolidated.  Multiple spaces in a row, for example, have the same effect as a single space, when IFS contains a space (or is not set at all).  Newlines also count as whitespace for this purpose, which has important ramifications when attempting to load an array with lines of input.+  * non-whitespace IFS characters are not ignored at the beginning and end of expansions, the way whitespace IFS characters are.
  
-Finally, we note that pathname expansion happens after word splitting, and can produce some very shocking results.+Whitespace IFS characters get consolidated. 
 + 
 +  * Multiple spaces in a row, for example, have the same effect as a single space, when IFS contains a space (or is not set at all). 
 +  * Newlines also count as whitespace for this purpose, which has important ramifications when attempting to load an array with lines of input. 
 + 
 +</WRAP> 
 + 
 +Pathname expansion happens after word splitting, and can produce some very shocking results:
  
 <code bash> <code bash>
Line 131: Line 231:
 qmaild:*:994:998::/var/qmail:/sbin/nologin qmaild:*:994:998::/var/qmail:/sbin/nologin
  
-IFS=:; args $(getent passwd qmaild)+IFS=:; test $(getent passwd qmaild)
 737 args: <qmaild> <00INDEX.lsof> <03> <037_ftpd.patch> ... 737 args: <qmaild> <00INDEX.lsof> <03> <037_ftpd.patch> ...
  
Line 137: Line 237:
 </code> </code>
  
-The * word, produced by the shell's word splitting, was then expanded as a glob, resulting in several hundred new and exciting words.  This can be disastrous if it happens unexpectedly.  As with most of the dangerous features of the shell, it is retained because "it's always worked that way".  In fact, it can be used for good, if you're very careful:+The **<nowiki>*</nowiki>** word, produced by the shell's word splitting, was then expanded as a glob, resulting in several hundred new and exciting words. 
 + 
 +  * This can be disastrous if it happens unexpectedly. 
 +  * As with most of the dangerous features of the shell, it is retained because "it's always worked that way". 
 +  * In fact, it can be used for good, if you're very careful:
  
 <code bash> <code bash>
 files='*.mp3 *.ogg' files='*.mp3 *.ogg'
-args $files+test.sh $files
 2 args: <Yello - Oh Yeah.mp3> <*.ogg> 2 args: <Yello - Oh Yeah.mp3> <*.ogg>
 </code> </code>
  
-Pathname expansion can be disabled with set -f or set -o noglob; though this can lead to surprising and confusing code.+<WRAP info> 
 +**NOTE:**  Pathname expansion can be disabled with set -f or set -o noglob; though this can lead to surprising and confusing code. 
 +</WRAP> 
  
 ---- ----
Line 153: Line 260:
   *  Word splitting is not performed on expansions inside Bash keywords such as [[ ... ]] and case.   *  Word splitting is not performed on expansions inside Bash keywords such as [[ ... ]] and case.
   *  Word splitting is not performed on expansions in assignments. Thus, one does not need to quote anything in a command like these:   *  Word splitting is not performed on expansions in assignments. Thus, one does not need to quote anything in a command like these:
- 
     * foo=$bar     * foo=$bar
- 
     * bar=$(a command)     * bar=$(a command)
- 
     * logfile=$logdir/foo-$(date +%Y%m%d)     * logfile=$logdir/foo-$(date +%Y%m%d)
- 
     * PATH=/usr/local/bin:$PATH ./myscript      * PATH=/usr/local/bin:$PATH ./myscript 
- 
   * In either case, quoting anyway will not break anything. So if in doubt, quote!   * In either case, quoting anyway will not break anything. So if in doubt, quote!
- 
   * When using the read command (which you generally don't want to use without -r), word splitting is performed on the input.  That's generally wanted when read -a is used (to populate an array) or when passing more than one variable to read.  Some amount of word splitting is still done when only one variable is given in that leading and trailing IFS whitespace characters are stripped and one trailing non-whitespace IFS character (possibly surrounded by IFS whitespace characters) would be stripped if there was no other separator in the input (like in IFS=": " read -r var <<< "input : ").  Quoting is irrelevant here, though this behavior can be disabled by emptying IFS, typically with IFS= read -r var   * When using the read command (which you generally don't want to use without -r), word splitting is performed on the input.  That's generally wanted when read -a is used (to populate an array) or when passing more than one variable to read.  Some amount of word splitting is still done when only one variable is given in that leading and trailing IFS whitespace characters are stripped and one trailing non-whitespace IFS character (possibly surrounded by IFS whitespace characters) would be stripped if there was no other separator in the input (like in IFS=": " read -r var <<< "input : ").  Quoting is irrelevant here, though this behavior can be disabled by emptying IFS, typically with IFS= read -r var
  
  
bash/wordsplitting.1610205307.txt.gz · Last modified: 2021/01/09 15:15 by peter
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki