Welcome to part 2 of Bash Shell Scripting at a beginner level. This article will dive into some more unique aspects of bash scripting. It will continue to use familiar commands, with an explain of anything new, and cover standard output standard input, standard error, the “pipe”, and data redirection.
Adding comments #
As your scripts get more complicated and functional you will need to add comments to remember what you were doing. If you share your scripts with others, comments will help them understand the thought process and what you intended for your script to do. From the last article recall there were mathematical equations. Some comments have been added in the new version. Notice that in the learnToScript.sh file (reproduced below) the comments are the lines with the hash sign before them. When the script runs these lines do not appear.
#!/bin/bash #Let's pick up from our last article. We #learned how to use mathematical equations #in bash scripting. echo $((5+3)) echo $((5-3)) echo $((5*3)) echo $((5/3))
[zexcon ~]$ ./learnToScript.sh 8 2 15 1
Pipe Operator |
We will use another tool called grep to introduce the pipe operator.
Grep searches one or more input files for lines containing a match to a specified pattern. By default, Grep outputs the matching lines.https://www.gnu.org/software/grep/
Paul W. Frields’ article in the Fedora Magazine provides a good background on grep.
You will find the pipe key above the Enter key. Enter it by pressing Shift + \. (English Keyboard)
Now that you are all freshened up on grep, look at an example of the use of the pipe command. At the command line type in ls -l | grep learn
[zexcon ~]$ ls -l | grep learn -rwxrw-rw-. 1 zexcon zexcon 70 Sep 17 10:10 learnToScript.sh
Normally the ls -l command would provide a list of the files on your screen. Here the full results of the ls -l command are piped into the grep command which searches for the string learn. Think of the pipe command like a filter. A command is run, in this case ls -l, and the results are limited to the files inside your directory. These results are sent via the pipe command to grep which searches for the work learn and only that line appears.
Look at one more example to try and nail this home. The less command will allow you to see the results of a command that would extend beyond one screen size. Here is a quick description from the man pages for the less command.
Less is a program similar to more(1), but which allows backward movement in the file as well asFedora 34 Manual(man) Pages
forward movement. Also, less does not have to read the entire input file before starting, so
with large input files it starts up faster than text editors like vi(1). Less uses termcap (or
terminfo on some systems), so it can run on a variety of terminals. There is even limited sup‐
port for hardcopy terminals. (On a hardcopy terminal, lines which should be printed at the top
of the screen are prefixed with a caret.)
So let’s see what it looks like utilizing the pipe and the less command
[zexcon ~]$ ls -l /etc | less
total 1504 drwxr-xr-x. 1 root root 126 Jul 7 17:46 abrt -rw-r--r--. 1 root root 18 Jul 7 16:04 adjtime -rw-r--r--. 1 root root 1529 Jun 23 2020 aliases drwxr-xr-x. 1 root root 70 Jul 7 17:47 alsa drwxr-xr-x. 1 root root 14 Apr 23 05:58 cron.d drwxr-xr-x. 1 root root 0 Jan 25 2021 cron.daily : :
The results have been trimmed, here, for readability. Use the arrow keys on the keyboard to scroll up or down. Unlike the command line, where you might miss the top of the results if they scroll off screen, you can control the display. To get out of the less screen tap the q key.
Standard Output (stdout), >, >>, 1>, and 1>>
The output of a command preceding the > or >> is sent to a file whose name follows. Keep in mind that > and 1> have the same results since the 1 stands for stdout (the standard output). Stdout is assumed if it does not appear. The >> and 1>> will append the data to the end of the file. In each case (> or >>) the file is created if it does not exist.
As an example, say you want to watch the ping command output to see if it dropped a packet. Rather than sit and watch the console, redirect the output to a file. You can come back later and see if packets were dropped. Here is a test of the redirect using >.
[zexcon ~]$ ls -l ~ > learnToScriptOutput
This takes the normal results you see in the terminal (recall ~ is your home directory) and redirects it to the learnToScriptOutput file. Did you notice that learnToScriptOutput was never created but now the file exists? Kind of cool.
total 128 drwxr-xr-x. 1 zexcon zexcon 268 Oct 1 16:02 Desktop drwxr-xr-x. 1 zexcon zexcon 80 Sep 16 08:53 Documents drwxr-xr-x. 1 zexcon zexcon 0 Oct 1 15:59 Downloads -rw-rw-r--. 1 zexcon zexcon 685 Oct 4 16:00 learnToScriptAllOutput -rw-rw-r--. 1 zexcon zexcon 23 Oct 4 12:42 learnToScriptInput -rw-rw-r--. 1 zexcon zexcon 0 Oct 4 16:42 learnToScriptOutput -rw-rw-r--. 1 zexcon zexcon 52 Oct 4 16:07 learnToScriptOutputError -rwxrw-rw-. 1 zexcon zexcon 477 Oct 4 15:01 learnToScript.sh drwxr-xr-x. 1 zexcon zexcon 0 Jul 7 16:04 Videos
Standard Error (stderr), 2>, and 2>>
The error output of a command preceding the > or >> is sent to a file whose name follows. Keep in mind that 2> and 2>> have the same result but the 2>> will append the data to the end of the file. So what is the purpose of these? What if you only want to catch an error. Then the 2> or 2>> is here to help. The 2 indicates the output that would normally go to stderr (standard error). Now put this into practice by listing a non-existent file.
[zexcon ~]$ ls -l /etc/invalidTest 2> learnToScriptOutputError
This takes the error results and redirects it to the learnToScriptOutputError file.
ls: cannot access '/etc/invalidTest': No such file or directory
All Output &>, &>> and |&
If you are thinking, I don’t want to write both standard output (stdout) and standard error (stderr) to different files. You are in luck. In Bash 5 the preferred way to redirect both stdout and stderr to the same file is to use &> or, as you might guess, &>> to append to a file.
[zexcon ~]$ ls -l ~ &>> learnToScriptAllOutput [zexcon ~]$ ls -l /etc/invalidTest &>> learnToScriptAllOutput
After running these commands, the output of both appear in the same file without identifying error or a standard output.
total 128 drwxr-xr-x. 1 zexcon zexcon 268 Oct 1 16:02 Desktop drwxr-xr-x. 1 zexcon zexcon 80 Sep 16 08:53 Documents drwxr-xr-x. 1 zexcon zexcon 0 Oct 1 15:59 Downloads -rw-rw-r--. 1 zexcon zexcon 685 Oct 4 16:00 learnToScriptAllOutput -rw-rw-r--. 1 zexcon zexcon 23 Oct 4 12:42 learnToScriptInput -rw-rw-r--. 1 zexcon zexcon 0 Oct 4 16:42 learnToScriptOutput -rw-rw-r--. 1 zexcon zexcon 52 Oct 4 16:07 learnToScriptOutputError -rwxrw-rw-. 1 zexcon zexcon 477 Oct 4 15:01 learnToScript.sh drwxr-xr-x. 1 zexcon zexcon 0 Jul 7 16:04 Videos ls: cannot access '/etc/invalidTest': No such file or directory
If you are working directly from the command line and looking to pipe all results to another command, you can use |& for this purpose.
[zexcon ~]$ ls -l |& grep learn -rw-rw-r--. 1 zexcon zexcon 1197 Oct 18 09:46 learnToScriptAllOutput -rw-rw-r--. 1 zexcon zexcon 343 Oct 14 10:47 learnToScriptError -rw-rw-r--. 1 zexcon zexcon 0 Oct 14 11:11 learnToScriptOut -rw-rw-r--. 1 zexcon zexcon 348 Oct 14 10:27 learnToScriptOutError -rwxr-x---. 1 zexcon zexcon 328 Oct 18 09:46 learnToScript.sh [zexcon ~]$
Standard Input (stdin)
You have used standard input (stdin) numerous times throughout articles 1 and 2 since your keyboard uses standard input every time you type a key. To give a bit of a change to the usual “it’s your keyboard”, let’s use the read command in a script. The read command, used in the script below, does what it sounds like, reads standard input.
#!/bin/bash #Here we are asking a question to prompt the user for standard input. i.e.keyboard echo 'Please enter your name.' #Here we are reading the standard input and assigning it to the variable name with the read command. read name #We are now going back to standard output, by using echo and printing your name to the command line. echo "With standard input you have told me your name is: $name"
This example prompts for input via standard output, for information it obtains from standard input(keyboard), storing it in a variable called name using read and displays the value in name via standard output.
[zexcon@fedora ~]$ ./learnToScript.sh Please enter your name. zexcon With standard input you have told me your name is: zexcon [zexcon@fedora ~]$
Into the script…
Now put what has been learned in a script to see how it can be used. The following is a new version of the previous learnToScript.sh file. There are a few added lines. It uses the append options for standard output, standard error and both into one file. It will write the standard output into learnToScriptStandardOutput, standard error into learnToScriptStandardError and both output and error into learnToScriptAllOutput
#!/bin/bash #As we know this article is about scripting. So let's #use what we learned in a script. #Let's get some information from the user and add it to our scripts with stanard input and read echo "What is your name? " read name #Here standard output directed to append a file to learnToScirptStandardOutput echo "$name, this will take standard output with append >> and redirect to learnToScriptStandardOutput." 1>> learnToScriptStandardOutput #Here we are taking the standard error and appending it to learnToScriptStandardError but to see this we need to #create an error. eco "Standard error with append >> redirect to learnToScriptStandardError." 2>> learnToScriptStandardError #Here we are going to create an error and a standard output and see they go to the same place. echo "Standard output with append >> redirect to learnToScriptAllOutput." &>> learnToScriptAllOutput eco "Standard error with append >> redirect to learnToScriptAllOutput." &>> learnToScriptAllOutput
This example creates three files in the same directory. The command echo is intentionally typed incorrectly to generate an error. If you check out all three files, you will see one message in learnToScriptStandardOutput, one in learnToScriptStandardError and two in learnToScriptAllOutput. Also notice the script prompts for a name which it writes to the learnToScriptStandardOutput.
At this point it should start to be clear that anything you can do on the command line you can also do in a script. When writing a script that others might use, documentation is extremely important. Continuing the dive into scripting, the standard output will make more sense as you will be the one generating them. Inside a script you can use the same things used from the command line. The next article will get into functions, loops and things that will continue to build on this foundation.
I recommend to use bash-modules library for scripting, which supports “unofficial strict mode for bash”, to convert simple bash scripts into command line utilities with proper option handling and error reporting.
See https://github.com/vlisivka/bash-modules .
Awesome. Thanks for sharing this.
Volodymyr this sounds interesting. I noticed it said it was “…developed at Fedora Linux…” so I attempted to find it in the repository but with no luck. Are you are the project owner?
Where is the first Part please? (i.e. Part #1)
Typo eco for echo. (Just before the conclusion
#Here are are taking the standard error and appending it to learnToScriptStandardError but to see this we need to #create an error.
eco “Standard error with append >> redirect to
Aside from the typo, A very very well done presentation.
I learned about &> and &>>.
No, this is done intentionally to generate an error to stderr, otherwise it will be empty.
But there are several typos in the article:
1) Here are are… (we are?)
2) Lets (Let’s)
3) informtion (information)
4) lack of the sign # in the beginning of each string of comments in several places
These were corrected
Yes, I’m owner of bash-modules project.
I would encourage in tutorials like this to use “set -euo pipefail” as the 2nd line of the script (perhaps this overlaps with Volodmyr’s bash-modules). In short this lets a script quit on first error, treat undefined variables are error (which they 99.999% are, see Steam’s installer script nuking home directory), and stopping a pipe chain to prevent errors/garbage being sent through them.