Differences

This shows you the differences between two versions of the page.

--- bash:files:read_a_file [2021/01/26 13:20] – peter
+++ bash:files:read_a_file [2021/01/26 14:19] (current) – [How to keep other commands from "eating" the input] peter
@@ Line 7: / Line 7: @@
 [[BASH:Files:Read a file:Basic read|Basic read]]
-----
+[[BASH:Files:Read a file:Input source selection|Input source selection]]
-===== Basic read =====
+[[BASH:Files:Read a file:Read fields from a file|Read fields from a file]]
-<code bash>
+[[BASH:Files:Read a file:Read from a command instead of a regular file|Read from a command instead of a regular file]]
-while read -r line; do
-  printf '%s\n' "$line"
-done < "$file"
-</code>
-<WRAP info>
+[[BASH:Files:Read a file:Read from an interactive shell|Read from an interactive shell]]
-**NOTE:**  This reads each line of the file into the **line** variable.
-  * **line**:  is a variable name, chosen by you.
+[[BASH:Files:Read a file:Skip Reading Comments|Skip Reading Comments]]
-  * **-r**:  Prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters).
-    * Without this option, any unescaped backslashes in the input will be discarded.
-    * You should almost always use the **-r** option with read.
-  * **< "$file"**:  The file to read.
-</WRAP>
-----
+[[BASH:Files:Read a file:Troubleshooting|Troubleshooting]]
-===== Prevent removal of leading and trailing white-space characters =====
-<code bash>
-while IFS= read -r line; do
-  printf '%s\n' "$line"
-done < "$file"
-</code>
-<WRAP info>
-**NOTE:**  Very similar to the basic read, but adding usage of **IFS**.
-The **IFS** (internal field separator) is often set to support reads.
-  * **IFS= **:  By default, read modifies each line read, by removing all leading and trailing white-space characters (spaces and tabs, if present in IFS, which is the default).
-    * To prevent this, the IFS variable is cleared.
-  * **line**:  is a variable name, chosen by you.
-  * **-r**:  Prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters).
-    * Without this option, any unescaped backslashes in the input will be discarded.
-    * You should almost always use the **-r** option with read.
-  * **< "$file"**:  The file to read.
-</WRAP>
 ----
-====== Read fields from a file ======
-To read fields within each line of the file, additional variables may be used with the read:
-For instance, if an input file has 3 columns separated by white space (space or tab characters only).
-<code bash>
-while read -r first_name last_name phone; do
-  # Only print the last name (second column).
-  printf '%s\n' "$last_name"
-done < "$file"
-</code>
-----
-If the field delimiters are not whitespace, set the IFS (internal field separator):
-<code bash>
-# Extract the username and its shell from /etc/passwd:
-while IFS=: read -r user pass uid gid gecos home shell; do
-  printf '%s: %s\n' "$user" "$shell"
-done < /etc/passwd
-</code>
-<WRAP info>
-**NOTE:**  IFS is set to **:**.
-For tab-delimited files, use **IFS=$'\t'** though beware that multiple tab characters in the input will be considered as one delimiter (and the **IFS=$'\t\t'** workaround will not work in Bash).
-You do not necessarily need to know how many fields each line of input contains.
-  * If you supply more variables than there are fields, the extra variables will be empty.
-  * If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied.
-For example:
-<code bash>
-read -r first last junk <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789'
-</code>
-  * **first**:   will contain "Bob"
-  * **last**:  will contain "Smith".
-  * **junk**:  holds everything else.
-The throwaway variable **_** can be used as a "junk variable" to ignore fields.
-  * It (or indeed any variable) can also be used more than once in a single read command, if we don't care what goes into it:
-<code bash>
-read -r _ _ first middle last _ <<< "$record"
-</code>
-  * We skip the first two fields, then read the next three.
-  * The final **_** can absorb any number of fields.
-    * It does not need to be repeated there.
-This usage of **_** is only guaranteed to work in Bash.
-  * Many other shells use _ for other purposes that will at best cause this to not have the desired effect, and can break the script entirely.
-  * It is better to choose a unique variable that isn't used elsewhere in the script, even though _ is a common Bash convention.
-</WRAP>
-----
-====== Field splitting, white-space trimming, and other input processing ======
-When not to use the **-r** option:
-  * **-r**:  Prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters).
-    * Without this option, any unescaped backslashes in the input will be discarded.
-    * You should almost always use the **-r** option with read.
-The most common exception to this rule is when **-e** is used, which uses Readline to obtain the line from an interactive shell.
-  * In that case, tab completion will add backslashes to escape spaces and such, and you do not want them to be literally included in the variable.
-  * This would never be used when reading anything line-by-line, though, and **-r** should always be used when doing so.
-----
-====== Skip Reading Comments ======
-To avoid reading comments starting with **#** simply skip them inside the loop:
-<code bash>
-while read -r line; do
-  [[ $line = \#* ]] && continue
-  printf '%s\n' "$line"
-done < "$file"
-</code>
-----
-====== Input source selection ======
-The redirection < "$file" tells the while loop to read from the file whose name is in the variable file.  If you would prefer to use a literal pathname instead of a variable, you may do that as well.  If your input source is the script's standard input, then you don't need any redirection at all.
-If your input source is the contents of a variable/parameter, bash can iterate over its lines using a here string:
-<code bash>
-while IFS= read -r line; do
-  printf '%s\n' "$line"
-done <<< "$var"
-</code>
-----
-The same can be done in any Bourne-type shell by using a "here document" (although read -r is POSIX, not Bourne):
-<code bash>
-while IFS= read -r line; do
-  printf '%s\n' "$line"
-done <<EOF
-$var
-EOF
-</code>
-----
-====== Read from a command instead of a regular file ======
-<code bash>
-some command | while IFS= read -r line; do
-  printf '%s\n' "$line"
-done
-</code>
-----
-This method is especially useful for processing the output of find with a block of commands:
-<code bash>
-find . -type f -print0 | while IFS= read -r -d '' file; do
-    mv "$file" "${file// /_}"
-done
-</code>
-**NOTE:**  This reads one filename at a time from the find command and renames the file, replacing spaces with underscores.
-  * **-print0**:  uses NUL bytes as filename delimiters.
-  * **-d ''**:  Instructs it to read all text into the file variable until it finds a NUL byte.
-    * By default, **find** and **read** delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behavior will split up those filenames at the newlines and cause the loop body to fail.
-    * **IFS= **:  Set to an empty string, because otherwise read would still strip leading and trailing whitespace.
-  * **|**:  Pipes the output from the find command into the while loop.
-    * This places the loop in a "sub shell", which means any state changes you make (changing variables, cd, opening and closing files, etc.) will be lost when the loop finishes.
-    * To avoid that, you may use a ProcessSubstitution:
-----
-<code bash>
-while IFS= read -r line; do
-  printf '%s\n' "$line"
-done < <(some command)
-</code>
-----
-===== My text files are broken!  They lack their final newlines! =====
-If there are some characters after the last line in the file (or to put it differently, if the last line is not terminated by a newline character), then read will read it but return false, leaving the broken partial line in the read variable(s).  You can process this after the loop:
-<code bash>
-# Emulate cat
-while IFS= read -r line; do
-  printf '%s\n' "$line"
-done < "$file"
-[[ -n $line ]] && printf %s "$line"
-</code>
-or:
-<code bash>
-# This does not work:
-printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done
-# This does not work either:
-printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line"
-# This works:
-printf 'line 1\ntruncated line 2' | { while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line"; }
-</code>
-The first example, beyond missing the after-loop test, is also missing quotes. See Quotes or Arguments for an explanation why. The Arguments page is an especially important read.
-For a discussion of why the second example above does not work as expected, see FAQ #24.
-----
-Alternatively, you can simply add a logical OR to the while test:
-<code bash>
-while IFS= read -r line || [[ -n $line ]]; do
-  printf '%s\n' "$line"
-done < "$file"
-printf 'line 1\ntruncated line 2' | while read -r line || [[ -n $line ]]; do echo "$line"; done
-</code>
-----
-===== How to keep other commands from "eating" the input =====
-Some commands greedily eat up all available data on standard input.  The examples above do not take precautions against such programs.  For example,
-<code bash>
-while read -r line; do
-  cat > ignoredfile
-  printf '%s\n' "$line"
-done < "$file"
-</code>
-will only print the contents of the first line, with the remaining contents going to "ignoredfile", as cat slurps up all available input.
-----
-One workaround is to use a numeric FileDescriptor rather than standard input:
-<code bash>
-# Bash
-while IFS= read -r -u 9 line; do
-  cat > ignoredfile
-  printf '%s\n' "$line"
-done 9< "$file"
-# Note that read -u is not portable to every shell. Use a redirect to ensure it works in any POSIX compliant shell:
-while IFS= read -r line <&9; do
-  cat > ignoredfile
-  printf '%s\n' "$line"
-done 9< "$file"
-</code>
-or:
-<code bash>
-exec 9< "$file"
-while IFS= read -r line <&9; do
-  cat > ignoredfile
-  printf '%s\n' "$line"
-done
-exec 9<&-
-</code>
-This example will wait for the user to type something into the file ignoredfile at each iteration instead of eating up the loop input.
-You might need this, for example, with mencoder which will accept user input if there is any, but will continue silently if there isn't.  Other commands that act this way include ssh and ffmpeg.  Additional workarounds for this are discussed in FAQ #89.