Bourne Shell Scripting/Redirection

From Wikibooks, the open-content textbooks collection

[edit] 'Standard out', 'Standard in' and 'Standard error'

These principles are really simple, but well worth knowing from the ground-up as they are some of the basic building blocks of shell-scripting.

Take the simple Bourne shell script below called 'hello.sh':

#!/bin/sh
echo Hello

And this is us running it from the '$' Unix prompt:

$ hello.sh
Hello
$


The line that reads '#!/bin/sh' just ensures that we are running in a Bourne shell - see previous chapter.

When we run the script, it simply outputs the string "Hello" to the screen and then returns us to our prompt.

This is probably normally what you want to do, however it is very easy to redirect this output to another device or file by using the '>' (greater-than) redirect operator like this:

$ hello.sh > myfile.txt
$


This time, we don't see the string 'Hello' on the screen. Where's it gone ?

Examine the file 'myfile.txt' using the 'cat' command:

$ cat myfile.txt
Hello
$


So what happened ?

When you run a shell-script under a Unix shell, all output is sent to a virtual destination known as the 'standard output'. By default your standard output will be your screen - but it doesn't have to be: you can re-route all output to a file, a physical device (such as a printer) or even to another process. Note than 'standard out' is commonly shortened to 'stdout'.

Run the program again, this time using the '>>' operator instead, and then examine 'myfile.txt' again using the 'cat' command:

$ hello.sh >> myfile.txt
$ cat myfile.txt
Hello
Hello
$


You can see that 'myfile.txt' now consists of two lines - the output has been added to the end of the file (or concatenated); this is due to the use of '>>' operator - if you run the script again , this time with the single greater-than operator, we get:

$ hello.sh > myfile.txt
$ cat myfile.txt
Hello
$


- Just one 'Hello' again - this is because the '>' will always wipe out the contents of an existing textfile if it already contains anything.

Ok - enough on 'stdout', lets move onto 'standard in' or 'stdin'.

By default 'stdin' is fed from your keyboard; run the 'cat' command without any arguments and it will just sit there, waiting for you type something:

$ cat


I can type all day here, and I never seem to get my prompt back from
this stupid machine.


I have even pressed RETURN a few times !!!
.....etc....etc


In fact 'cat' will sit there all day until you type a 'CTRL-D' (the 'End of File Character' or 'EOF' for short). This seems to catch out a lot of people new to using cat.

To redirect our 'stdin' from somewhere else - use the '<' (less-than arrow):

$ cat < myfile.txt
Hello
$

So 'cat' will now read from the text file 'myfile.txt' - the 'EOF' character is also generated at the end of file, so 'cat' will exit as before.

NB: Previously we used 'cat' in this format:

cat myfile.txt

Which is functionally identical to

cat < myfile.txt

However, this is using two fundamentally different mechanisms - one uses an argument to the command, the other is more general and redirects 'stdin' - which is what we're concerned with here. It's more convenient to use 'cat' with a filename as argument - that's why the inventors of 'cat' put this in, however not all programs and scripts are going to take arguments, so this is just an easy example.

It's possible to redirect 'stdin' and 'stdout' in one line:

$ cat < myfile.txt > mynewfile.txt


Will copy the contents of 'myfile.txt' to 'mynewfile.txt' (and will overwrite any previous content in 'mynewfile.txt'. (Again we could have (and would have done in real life) used 'cp myfile.txt mynewfile.txt'.)

Or this:

$ cat myfile.txt >> mynewfile.txt


Which would append the contents of 'myfile.txt' to the end of 'mynewfile.txt'.

There is another stream in addition to 'stdin' and 'stdout' - which is known as 'standard error' - or 'stderr', which is useful for capturing error messages generated by programs to logfiles for instance.

As an example if you run the command 'ls myfile.txt', it simply lists the filename 'myfile.txt' - if it exists. If the file 'myfile.txt' does NOT exist, 'ls' will return an error to the 'stderr' stream - which by default is also your screen.

So, lets run 'ls' a couple of times, first on a file which does exist and then on one that doesn't:

$ ls myfile.txt
myfile.txt
$


and then:

$ ls nosuchfile.txt
ls: no such file or directory
$


And again, this time with 'stdout' redirected only:

$ ls nosuchfile.txt > logfile.txt
ls: no such file or directory
$


We still see the error message - 'logfile.txt' will be created - but empty.

Once more, this time we will use the '>' operator preceded by the stream number 2: (which corresponds to 'stderr'):

$ ls nosuchfile.txt 2> logfile.txt
$


No output to the screen, but if we examine 'logfile.txt':

$ cat logfile.txt
ls: no such file or directory
$


We can also redirect both 'stdout' and 'stderr' independently like this:

$ ls nosuchfile.txt > stdio.txt 2>logfile.txt
$


'stdio.txt' will be blank , 'logfile.txt' will contain the error as before.

Sometimes it's desirable to send both 'stdout' and 'stderr' to the same location, this can be done using the shorthand:

$ ls nosuchfile.txt > alloutput.txt 2>&1


Where '2>&1' means something like 'stderr should follow stdout, wherevever thats being redirected to.'

[edit] Pipes, Tees and Named Pipes

Now we come to a very useful mechanism - redirection by using a 'pipe' - represented by the pipe symbol '|'. The pipe allows you to join up commands so that you 'pipe' the stdout of one command directly into the stdin of another.

So for instance the 'grep' command (which returns matching string, given a keyword and some text to search), can be 'piped-into' from the result of another command. A common usage of this is to look for running processes on a machine using the 'ps' command.

If you give the command

$ ps -eaf


it will generally list pagefuls of running processes on your machine, which you would have to sift through manually to find what you want. Let's say you are looking for a process which you know contains the word 'oracle'; use the output of 'ps' to pipe into grep, which will only return the matching lines:

$ ps -eaf | grep oracle


Now you will only get back the lines you need; what happens if there's loads of these still? No problem, pipe the output to the command 'more' (or 'pg') - which will pause your screen if it fills up:

$ ps -ef | grep oracle | more


Pipes can be used to link as many programs as you wish within reasonable limits (don't know what these limits are!)

Don't forgot you can still use the redirectors in combination: $ ps -ef | grep oracle > /tmp/myprocesses.txt


There is another useful mechanism that can be used with pipes - that is the 'tee'; imagine a pipe shaped like a 'T' - one input, two outputs:

$ ps -ef | grep oracle | tee /tmp/myprocesses.txt


The 'tee' will copy whatever's given to its stdin and redirect this to the argument given (a file); it will also then send a further copy to its stdout - which means you can effectively intercept the pipe, take a copy at this stage, and carry on piping up other commands; useful maybe for outputting to a logfile, and copying to the screen.

There is a variation on the in-line pipe which we have been discussing called the 'named pipe'. A named pipe, is actually a file, with its own 'stdin' and 'stdout' - which you attach processes to.

This is useful for programs to talk to each other, especially when you don't know when one program will try and talk to the other (waiting for a backup to finish etc) and when you don't want to write a complicated network-based listener, or do a clumsy polling loop:

To create a 'named pipe', you need to run the 'mkfifo' (fifo=first in, first out; so data is read out in the same order as it is written into).

$ mkfifo mypipe
$


A named pipe called 'mypipe' is created; now we can start using it:

This test is best run with two terminals logged in:

1. From 'terminal a'
$ cat < mypipe


The 'cat' will sit there waiting for an input.

2. From 'terminal b'
$ cat myfile.txt > mypipe
$


This should finish immediately. Flick back to 'terminal a'; this will now have read from the pipe and received an 'EOF', and you will see the data on the screen; the command will have finished, and you are back at the command prompt.

Now try the other way round:

1. From terminal 'b'
$ cat myfile.txt > mypipe
This will now sit there, as there isn't another process on the other end to 'drain' the pipe - it's blocked.

2. From terminal 'a'
$ cat < mypipe


As before, both processes will now finish, the output showing on terminal 'a'.

It's worth pointing out here, that since Unix generally abstracts physical devices as files; so redirects and pipes are useful in this domain also; for instance (in the 'Solaris' flavour of Unix, the loudspeaker and it's microphone can be accessed by the file '/dev/audio').

So :
# cat /tmp/audio.au > /dev/audio
Will play a sound...

whereas:
# cat < /dev/audio > /tmp/mysound.au


Will record a sound.(you will need to CTRL-C this to finish...)

This is fun:
# cat < /dev/audio > /dev/audio


Now wave the microphone around whilst shouting - Jimi Hendrix style feedback. Great stuff. You will probably need to be logged in as 'root' to try this by the way.