Wiki

User Tools

Site Tools

Trace: • bild_de • verylong.cpp • python • sftp • dropbox • networking • files_system • database_api • arcade_games • price_action
bash:globs

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
bash:globs [2019/12/07 12:42] – peterbash:globs [2021/02/04 09:43] (current) – [Glob Star - globstar] peter
Line 1: Line 1:
 ====== BASH - Globs ====== ====== BASH - Globs ======
  
-"Glob" is the common name for a set of Bash features that match or expand specific types of patterns.+**Glob** or **Globstar** is the common name for a set of Bash features that match or expand specific types of patterns.
  
 Some synonyms for globbing (depending on the context in which it appears) are pattern matching, pattern expansion, filename expansion, and so on. Some synonyms for globbing (depending on the context in which it appears) are pattern matching, pattern expansion, filename expansion, and so on.
Line 7: Line 7:
 A glob may look like ***.txt** and, when used to match filenames, is sometimes called a "wildcard". A glob may look like ***.txt** and, when used to match filenames, is sometimes called a "wildcard".
  
-Traditional shell globs use a very simple syntax, which is less expressive than a RegularExpression.+Traditional shell globs use a very simple syntax, which is less expressive than a Regular Expression.
  
 Most characters in a glob are treated literally, but a * matches 0 or more characters, a ? matches precisely one character, and [...] matches any single character in a specified set (see Ranges below). Most characters in a glob are treated literally, but a * matches 0 or more characters, a ? matches precisely one character, and [...] matches any single character in a specified set (see Ranges below).
Line 35: Line 35:
 tar xvf *.tar tar xvf *.tar
 # Expands to: tar xvf file1.tar file2.tar file42.tar ... # Expands to: tar xvf file1.tar file2.tar file42.tar ...
-# (which is generally not what one wants) 
 </code> </code>
  
Line 49: Line 48:
 # This is safe even if a filename contains whitespace: # This is safe even if a filename contains whitespace:
 for f in *.tar; do for f in *.tar; do
-    tar tvf "$f"+  tar tvf "$f"
 done done
  
 # But this one is not: # But this one is not:
 for f in $(ls | grep '\.tar$'); do for f in $(ls | grep '\.tar$'); do
-    tar tvf "$f"+  tar tvf "$f"
 done done
 </code> </code>
  
-In the second example above, the output of **ls** is filtered, and then the result of the whole pipeline is divided into words, to serve as iterative values for the loop.+<WRAP info> 
 +**NOTE:**  In the second example above, the output of **ls** is filtered, and then the result of the whole pipeline is divided into words, to serve as iterative values for the loop.
  
 This word-splitting will occur at internal whitespace within each filename, which makes it useless in the general case. This word-splitting will occur at internal whitespace within each filename, which makes it useless in the general case.
Line 65: Line 65:
  
 For more such examples, see BashPitfalls. For more such examples, see BashPitfalls.
 +
 +</WRAP>
  
 ---- ----
Line 76: Line 78:
 <code bash> <code bash>
 case "$input" in case "$input" in
-    [Yy]|'') confirm=1;; +  [Yy]|'') confirm=1;; 
-    [Nn]*) confirm=0;; +  [Nn]*) confirm=0;; 
-    *) echo "I don't understand.  Please try again.";;+  *) echo "I don't understand.  Please try again.";;
 esac esac
 </code> </code>
  
-Patterns (which are separated by | characters) are matched against the first word after the case itself.+<WRAP info> 
 +**NOTE:**  Patterns (which are separated by | characters) are matched against the first word after the case itself.
  
 The first pattern which matches, "wins", causing the corresponding commands to be executed. The first pattern which matches, "wins", causing the corresponding commands to be executed.
 +
 +</WRAP>
  
 ---- ----
Line 128: Line 133:
 |<nowiki>[[:digit:]_.]</nowiki>|Matches any digit, or _ or .| |<nowiki>[[:digit:]_.]</nowiki>|Matches any digit, or _ or .|
  
-In most shell implementations, one may also use ^ as the range negation character, e.g. [^[:space:]].+<WRAP info> 
 +**NOTE:**  In most shell implementations, one may also use ^ as the range negation character, e.g. [^[:space:]].
  
 However, POSIX specifies ! for this role, and therefore ! is the standard choice. However, POSIX specifies ! for this role, and therefore ! is the standard choice.
  
 Recent Bash versions Interpret [a-d] as [abcd].  To match a literal **-**, include it as first or last character. Recent Bash versions Interpret [a-d] as [abcd].  To match a literal **-**, include it as first or last character.
 +
 +</WRAP>
  
 ---- ----
  
-===== Options which change globbing behaviour =====+===== Options which change globbing behavior =====
  
-==== extglob ====+==== Extended Globs - extglob ====
  
 In addition to the traditional globs (supported by all Bourne-family shells) that we've seen so far, Bash (and Korn Shell) offers extended globs, which have the expressive power of regular expressions. In addition to the traditional globs (supported by all Bourne-family shells) that we've seen so far, Bash (and Korn Shell) offers extended globs, which have the expressive power of regular expressions.
Line 163: Line 171:
 </code> </code>
  
 +<WRAP info>
 +**NOTE:**  Patterns in a list are separated by | characters.
 +</WRAP>
  
-Patterns in a list are separated by | characters. 
  
-Extended globs allow you to solve a number of problems which otherwise require a rather surprising amount of ugly hacking; for example,+----
  
-Toggle line numbers+==== Example of using Extended Globs ====
  
 +Extended globs allow you to solve a number of problems which otherwise require a rather surprising amount of ugly hacking; for example,
 +
 +<code bash>
 # To remove all the files except ones matching *.jpg: # To remove all the files except ones matching *.jpg:
 rm !(*.jpg) rm !(*.jpg)
 +
 # All except *.jpg and *.gif and *.png: # All except *.jpg and *.gif and *.png:
 rm !(*.jpg|*.gif|*.png) rm !(*.jpg|*.gif|*.png)
 +</code>
  
-Toggle line numbers+or
  
-# To copy all the MP3 songs except one to your device+<code bash> 
 +# To copy all the MP3 songs except one to your device.
 cp !(04*).mp3 /mnt cp !(04*).mp3 /mnt
 +</code>
  
-To use an extglob in a parameter expansion (this can also be done in one BASH statement with read):+----
  
-Toggle line numbers+==== Extended Globs with Parameter Expansion ====
  
-# To trim leading and trailing whitespace from a variable+To use an extglob in a parameter expansion (this can also be done in one BASH statement with read): 
 + 
 +<code bash> 
 +# To trim leading and trailing whitespace from a variable.
 x=${x##+([[:space:]])}; x=${x%%+([[:space:]])} x=${x##+([[:space:]])}; x=${x%%+([[:space:]])}
 +</code>
  
-Extended glob patterns can be nested, too.+----
  
-Toggle line numbers+==== Nested Extended Glob Patterns ====
  
 +Extended glob patterns can be nested, too.
 +
 +<code bash>
 [[ $fruit = @(ba*(na)|a+(p)le) ]] && echo "Nice fruit" [[ $fruit = @(ba*(na)|a+(p)le) ]] && echo "Nice fruit"
 +</code>
  
-extglob changes the way certain characters are parsed. It is necessary to have a newline (not just a semicolon) between shopt -s extglob and any subsequent commands to use it. You cannot enable extended globs inside a group command that uses them, because the entire block is parsed before the shopt is evaluated. Note that the typical function body is a group command. An unpleasant workaround could be to use a subshell command list as the function body. 
  
-Therefore, if you use this option in a script, it is best put right under the shebang line. 
  
-Toggle line numbers+<WRAP info> 
 +**NOTE:**  extglob changes the way certain characters are parsed.
  
 +It is necessary to have a newline (not just a semicolon) between **shopt -s extglob** and any subsequent commands to use it.
 +
 +You cannot enable extended globs inside a group command that uses them, because the entire block is parsed before the **shopt** is evaluated.
 +
 +The typical function body is a group command.  An unpleasant workaround could be to use a subshell command list as the function body.
 +</WRAP>
 +
 +Therefore, if you use this option in a script, it is best put right under the shebang line.
 +
 +<code bash>
 #!/usr/bin/env bash #!/usr/bin/env bash
-shopt -s extglob   # and others, such as nullglob dotglob+shopt -s extglob   # and others, such as nullglob dotglob. 
 +</code>
  
 If your code must be sourced and needs extglob, ensure it preserves the original setting from your shell: If your code must be sourced and needs extglob, ensure it preserves the original setting from your shell:
  
-Toggle line numbers +<code bash> 
- +# Remember whether extglob was originally set, so we know whether to unset it.
-# remember whether extglob was originally set, so we know whether to unset it+
 shopt -q extglob; extglob_set=$? shopt -q extglob; extglob_set=$?
-# set extglob if it wasn't originally set.+# Set extglob if it wasn't originally set.
 ((extglob_set)) && shopt -s extglob ((extglob_set)) && shopt -s extglob
 # Note, 0 (true) from shopt -q is "false" in a math context. # Note, 0 (true) from shopt -q is "false" in a math context.
  
 # The basic concept behind the following is to delay parsing of the globs until evaluation. # The basic concept behind the following is to delay parsing of the globs until evaluation.
-# This matters at group commands, such as functions in { } blocks+# This matters at group commands, such as functions in { } blocks.
  
 declare -a s='( !(x) )' declare -a s='( !(x) )'
Line 222: Line 256:
 eval 'echo !(x)'  # using eval if no other option. eval 'echo !(x)'  # using eval if no other option.
  
-# unset extglob if it wasn't originally set+# Unset extglob if it wasn't originally set.
 ((extglob_set)) && shopt -u extglob ((extglob_set)) && shopt -u extglob
  
 This should also apply for other shell options. This should also apply for other shell options.
 +</code>
  
-nullglob+---- 
 + 
 +===== Null Glob - nullglob =====
  
 nullglob expands non-matching globs to zero arguments, rather than to themselves. nullglob expands non-matching globs to zero arguments, rather than to themselves.
  
-Toggle line numbers +<code bash>
 $ ls *.c $ ls *.c
 ls: cannot access *.c: No such file or directory ls: cannot access *.c: No such file or directory
  
-# with nullglob set+# With nullglob set.
 shopt -s nullglob shopt -s nullglob
 ls *.c ls *.c
 # Runs "ls" with no arguments, and lists EVERYTHING # Runs "ls" with no arguments, and lists EVERYTHING
 +</code>
  
 Typically, nullglob is used to count the number of files matching a pattern: Typically, nullglob is used to count the number of files matching a pattern:
  
-Toggle line numbers +<code bash>
 shopt -s nullglob shopt -s nullglob
 files=(*) files=(*)
 echo "There are ${#files[@]} files in this directory." echo "There are ${#files[@]} files in this directory."
 +</code>
  
 Without nullglob, the glob would expand to a literal * in an empty directory, resulting in an erroneous count of 1. Without nullglob, the glob would expand to a literal * in an empty directory, resulting in an erroneous count of 1.
  
-Warning+----
  
-Enabling nullglob on a wide scope can trigger bugs caused by bad programming practices. It "breaks" the expectations of many utilities.+==== Null Blob BUG ====
  
-Removing array elements:+<WRAP important> 
 +**WARNING:**  Enabling nullglob on a wide scope can trigger bugs caused by bad programming practices.
  
-Toggle line numbers+It "breaks" the expectations of many utilities.
  
 +Removing array elements:
 +
 +<code bash>
 shopt -s nullglob shopt -s nullglob
 unset array[1] unset array[1]
 #unsets nothing #unsets nothing
 +
 unset -v "array[1]" unset -v "array[1]"
 #correct #correct
 +</code>
  
 Array member assignments in compound form using subscripts: Array member assignments in compound form using subscripts:
  
-Toggle line numbers +<code bash>
 shopt -s nullglob shopt -s nullglob
 array=([1]=*) array=([1]=*)
-#results in an empty array+# Results in an empty array. 
 +</code>
  
-This was reported as a bug in 2012, yet is unchanged to this day.+This was reported as a [[http://lists.gnu.org/archive/html/bug-bash/2012-08/msg00032.html|bug]] in 2012, yet is unchanged to this day. 
 +</WRAP>
  
 Apart from few builtins that use modified parsing under special conditions (e.g. declare) always use Quotes when arguments to simple commands could be interpreted as globs. Apart from few builtins that use modified parsing under special conditions (e.g. declare) always use Quotes when arguments to simple commands could be interpreted as globs.
  
-Enabling failglob, nullglob, or both during development and testing can help catch mistakes early. To prevent pathname expansion occuring in unintended places, you can set failglob. However, you must then guarantee all intended globs match at least one file. Also note that the result of a glob expansion does not always differ from the glob itself. failglob won't distinguish echo ? from echo '?' in a directory containing only a file named ?. nullglob will.+Enabling failglob, nullglob, or both during development and testing can help catch mistakes early.
  
-Portability+To prevent pathname expansion occurring in unintended places, you can set **failglob**.  However, you must then guarantee all intended globs match at least one file.  Also note that the result of a glob expansion does not always differ from the glob itself.  failglob won't distinguish echo ? from echo '?' in a directory containing only a file named ?.  nullglob will.
  
-"null globbing" is not specified by POSIX. In portable scripts, you must explicitly check that a glob match was successful by checking that the files actually exist. 
  
-Toggle line numbers+----
  
 +==== Null Glob Portability ====
 +
 +"null globbing" is not specified by POSIX.
 +
 +In portable scripts, you must explicitly check that a glob match was successful by checking that the files actually exist.
 +
 +<code bash>
 # POSIX # POSIX
  
 for x in *; do for x in *; do
-    [ -e "$x" ] || break +  [ -e "$x" ] || break 
-    ...+  ...
 done done
  
 f() { f() {
-    [ -e "$1" ] || return 1+  [ -e "$1" ] || return 1
  
-    for x do +  for x do 
-        ... +    ... 
-    done+  done
 } }
  
 f * || echo "No files found" f * || echo "No files found"
 +</code>
  
 Some modern POSIX-compatible shells allow null globbing as an extension. Some modern POSIX-compatible shells allow null globbing as an extension.
  
-Toggle line numbers +<code bash>
 # Bash # Bash
 shopt -s nullglob shopt -s nullglob
 +</code>
  
-In ksh93, there is no toggle-able option. Rather, that the "nullglob" behavior is to be enabled is specified inline using the "N" option to the ∼() sub-pattern syntax. +In ksh93, there is no toggle-able option.  Rather, that the "nullglob" behavior is to be enabled is specified inline using the "N" option to the ∼() sub-pattern syntax.
- +
-Toggle line numbers+
  
 +<code bash>
 # ksh93 # ksh93
  
 for x in ~(N)*; do for x in ~(N)*; do
-    ...+  ...
 done done
 +</code>
  
 In zsh, an toggle-able option(NULL_GLOB) or a glob qualifier(N) can be used. In zsh, an toggle-able option(NULL_GLOB) or a glob qualifier(N) can be used.
  
-Toggle line numbers +<code bash>
 # zsh # zsh
 for x in *(N); do ...; done # or setopt NULL_GLOB for x in *(N); do ...; done # or setopt NULL_GLOB
 +</code>
  
 mksh doesn't yet support nullglob (maintainer says he'll think about it). mksh doesn't yet support nullglob (maintainer says he'll think about it).
  
-dotglob+----
  
-By convention, a filename beginning with a dot is "hidden", and not shown by ls. Globbing uses the same convention -- filenames beginning with a dot are not matched by a glob, unless the glob also begins with a dot. Bash has a dotglob option that lets globs match "dot files":+===== Dot Glob - dotglob =====
  
-Toggle line numbers+By convention, a filename beginning with a dot is "hidden", and not shown by ls.
  
 +Globbing uses the same convention -- filenames beginning with a dot are not matched by a glob, unless the glob also begins with a dot.
 +
 +Bash has a dotglob option that lets globs match "dot files":
 +
 +<code bash>
 shopt -s dotglob nullglob shopt -s dotglob nullglob
 files=(*) files=(*)
 echo "There are ${#files[@]} files here, including dot files and subdirs" echo "There are ${#files[@]} files here, including dot files and subdirs"
 +</code>
  
-It should be noted that when dotglob is enabled, * will match files like .bashrc but not the . or .. directories. This is orthogonal to the problem of matching "just the dot files" -- a glob of .* will match . and .., typically causing problems.+It should be noted that when dotglob is enabled, * will match files like .bashrc but not the . or .. directories.
  
-globstar (since bash 4.0-alpha)+This is orthogonal to the problem of matching "just the dot files" -- a glob of .* will match . and .., typically causing problems.
  
-globstar recursively repeats a pattern containing '**'.+----
  
-Toggle line numbers+===== Glob Star - globstar =====
  
-$ shopt -s globstar; tree+(since bash 4.0-alpha) 
 + 
 +globstar recursively repeats a pattern containing **<nowiki>**</nowiki>**. 
 + 
 +<code bash> 
 +shopt -s globstar; tree
 . .
 ├── directory2 ├── directory2
Line 354: Line 416:
 ├── file1 ├── file1
 └── file2.c └── file2.c
 +</code>
  
-# Suppose that for the following examples.+Suppose that for the following examples.
  
 Matching files: Matching files:
  
-Toggle line numbers +<code bash>
 $ files=(**) $ files=(**)
 # equivalent to: files=(* */* */*/*) # equivalent to: files=(* */* */*/*)
Line 370: Line 432:
 # corresponds to: find -name "*.c" # corresponds to: find -name "*.c"
 # Caveat: **.c will not work, as it expands to *.c/*.c/… # Caveat: **.c will not work, as it expands to *.c/*.c/…
 +</code>
  
-Just like '*', '**' followed by a '/' will only match directories:+<WRAP info> 
 +**NOTE:**  To disable globstar use
  
-Toggle line numbers+<code bash> 
 +shopt -u globstar 
 +</code>
  
-$ files=(**/) +See:  **help shopt** for details. 
-# finds all subdirectories+</WRAP>
  
-$ files=(. **/) 
-# finds all subdirectories, including the current directory 
-# corresponds to: find -type d 
  
-failglob+----
  
-If a pattern fails to match, bash reports an expansion error. This can be useful at the commandline:+Assume you have a folder structure:
  
-Toggle line numbers+<code> 
 +. 
 +├── bar 
 +│   ├── foo 
 +│   │   └── baz 
 +│   │       └── hurz 
 +│   │           └── lolz 
 +│   │               └── hello.txt 
 +│   └── poit.txt 
 +└── fnord.txt 
 +</code>
  
 +Then **ls** with single star **<nowiki>*</nowiki>** would list:
 +
 +<code bash>
 +ls *.txt
 +fnord.txt
 +</code>
 +
 +The double star operator **<nowiki>**</nowiki>** will work on the subfolders.  The output will look like:
 +
 +<code bash>
 +ls **/*.txt
 +bar/foo/baz/hurz/lolz/hello.txt  bar/poit.txt  fnord.txt
 +</code>
 +
 +----
 +
 +<WRAP info>
 +**NOTE:**  Just like **<nowiki>*</nowiki>**, **<nowiki>**</nowiki>** followed by a **<nowiki>/</nowiki>** will only match directories:
 +</WRAP>
 +
 +
 +<code bash>
 +files=(**/)
 +# Finds all subdirectories.
 +
 +files=(. **/)
 +# Finds all subdirectories, including the current directory.
 +# Corresponds to: find -type d.
 +</code>
 +
 +----
 +
 +===== Fail Glob - failglob =====
 +
 +If a pattern fails to match, bash reports an expansion error.
 +
 +This can be useful at the commandline:
 +
 +<code bash>
 # Good at the command line! # Good at the command line!
 $ > *.foo # creates file '*.foo' if glob fails to match $ > *.foo # creates file '*.foo' if glob fails to match
Line 393: Line 505:
 $ > *.foo # doesn't get executed $ > *.foo # doesn't get executed
 -bash: no match: *.foo -bash: no match: *.foo
 +</code>
  
-GLOBIGNORE+----
  
-The Bash variable (not shopt) GLOBIGNORE allows you to specify patterns a glob should not match. This lets you work around the infamous "I want to match all of my dot files, but not . or .." problem:+===== GLOBIGNORE =====
  
-Toggle line numbers+The Bash variable (not shopt) GLOBIGNORE allows you to specify patterns a glob should not match.
  
 +This lets you work around the infamous "I want to match all of my dot files, but not . or .." problem:
 +
 +<code bash>
 $ echo .* $ echo .*
 . .. .bash_history .bash_logout .bashrc .inputrc .vimrc . .. .bash_history .bash_logout .bashrc .inputrc .vimrc
Line 405: Line 521:
 $ echo .* $ echo .*
 .bash_history .bash_logout .bashrc .inputrc .vimrc .bash_history .bash_logout .bashrc .inputrc .vimrc
 +</code>
  
-Unset GLOBIGNORE+----
  
-Toggle line numbers+==== Unset GLOBIGNORE ====
  
 +<code bash>
 $ GLOBIGNORE= $ GLOBIGNORE=
 $ echo .* $ echo .*
 . .. .bash_history .bash_logout .bashrc .inputrc .vimrc . .. .bash_history .bash_logout .bashrc .inputrc .vimrc
 +</code>
  
-nocasematch 
  
-Globs inside [[ and case commands are matched case-insensitive:+----
  
-Toggle line numbers+===== No Case Match - nocasematch =====
  
 +Globs inside <nowiki>[[</nowiki> and case commands are matched case-insensitive:
 +
 +<code bash>
 foo() { foo() {
-   local f r=0 nc=0 +  local f r=0 nc=0 
-   shopt -q nocasematch && nc=1 || shopt -s nocasematch +  shopt -q nocasematch && nc=1 || shopt -s nocasematch 
-   for f; do +  for f; do 
-      [[ $f = *.@(txt|jpg) ]] || continue +    [[ $f = *.@(txt|jpg) ]] || continue 
-      cmd -on "$f" || r=1 +    cmd -on "$f" || r=1 
-   done +  done 
-   ((nc)) || shopt -u nocasematch +  ((nc)) || shopt -u nocasematch 
-   return $r+  return $r
 } }
 +</code>
  
 This is conventionally done this way: This is conventionally done this way:
  
-Toggle line numbers +<code bash>
 case $f in case $f in
-    *.[Tt][Xx][Tt]|*.[Jj][Pp][Gg]) : ;; +  *.[Tt][Xx][Tt]|*.[Jj][Pp][Gg]) : ;; 
-    *) continue+  *) continue
 esac esac
 +</code>
  
 and in earlier versions of bash we'd use a similar glob: and in earlier versions of bash we'd use a similar glob:
  
-Toggle line numbers +<code bash>
 [[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp][Gg]) ]] || continue [[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp][Gg]) ]] || continue
 +</code>
  
 or with no extglob: or with no extglob:
  
-Toggle line numbers +<code bash>
 [[ $f = *.[Tt][Xx][Tt] ]] || [[ $f = *.[Jj][Pp][Gg] ]] || continue [[ $f = *.[Tt][Xx][Tt] ]] || [[ $f = *.[Jj][Pp][Gg] ]] || continue
 +</code>
  
-Here, one might keep the tests separate for maintenance; they can be easily reused and dropped, +Here, one might keep the tests separate for maintenance; they can be easily reused and dropped, without having to concern oneself with where they fit in relation to an internal ||. 
- +
-    without having to concern oneself with where they fit in relation to an internal ||. +
  
 Note also: Note also:
  
-Toggle line numbers +<code bash>
 [[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp]?([Ee])[Gg]) ]] [[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp]?([Ee])[Gg]) ]]
 +</code>
  
 Variants left as an exercise. Variants left as an exercise.
  
-nocaseglob (since bash 2.02-alpha1)+---- 
 + 
 +===== No Case Glob - nocaseglob ===== 
 + 
 +(since bash 2.02-alpha1) 
 + 
 +This option makes pathname expansion case-insensitive.
  
-This option makes pathname expansion case-insensitive. In contrast, nocasematch operates on matches in <nowiki>[[</nowiki> and case commands. +In contrast, nocasematch operates on matches in <nowiki>[[</nowiki> and case commands. 
  
bash/globs.1575722524.txt.gz · Last modified: 2020/07/15 09:30 (external edit)
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki