OpenSSH/Cookbook/Remote Processes

From Wikibooks, open books for an open world
Jump to: navigation, search

One of the main functions of OpenSSH is that of accessing and running programs on other systems. In addition to an interactive login, ssh can be used to simply execute a program or script. Logout is automatic when the program or script has run its course. Some combinations are readily obvious. Others require more careful planning. Sometimes it is enough of a clue just to know that something can be done, at other times more detail is required. A number of examples of useful combinations of using OpenSSH to run remote tasks follow.

Run a Remote Process[edit]

An obvious use of ssh(1) is to use the shell interactively. That is, after all, one of the main purposes of the program. There are several ways to expand upon that, either interactively or as part of unattended scripts. ssh(1) returns whatever exit value was returned by the remote process. When a remote process is completed, ssh(1) will terminate and pass on the exit value of the last remote process to complete. So in this way, it can be used in scripts and the outcome of the remote processes can be used.

The following will return success, 0, on the local system where ssh(1) was run.

ssh -l fred server.example.org /bin/true
echo $?

The following will return failure, 1, on the local system where ssh(1) was run.

ssh -l fred server.example.org /bin/false
echo $?

If any other values, from 0 to 255, were returned, ssh(1) will pass them on.

Run a remote process and capture output locally[edit]

Output from programs run on the remote machine can be saved locally using a normal redirect. Here we run dmesg(8) on the remote machine:

ssh -l fred server.example.org dmesg > /tmp/dmesg.from.server.log

Interactive processes will be difficult or impossible to operate in that manner because no output will be seen. For interactive processes requiring any user input, output can be piped through tee instead to send the output both to the file and to stdout. This example runs a FTP session remotely and logs the output locally.

ssh -l fred server.example.org "ftp -a anotherserver" | tee /tmp/ftp.log

It may be necessary to force pseudo-tty allocation to get both input and output to be properly visible.

ssh -t -l fred server.example.org "ftp -a anotherserver" | tee /tmp/ftp.log

Run a remote process in the background while disconnected[edit]

Many routine tasks can be set in motion and then left to complete on their own without needing to stay logged in. When running remote process in background it is useful to spawn a shell just for that task.

$ ssh -t -l fred server.example.org 'sh -c "tar zcf /backup/usr.tgz /usr/" &' 

Another way is to use a terminal multiplexer, like tmux(1) or screen(1), to be able to reconnect and follow the process from time to time.

# start session and go about business
$ ssh -l fred server.example.org screen

Once a tmux(1) or screen(1) session is running, it is possible to detach it and close the ssh connection without disturbing the background processes it may be running.

# start session and pick up (reattach) session in progress
$ ssh -l fred server.example.org "screen -x"  ???

More on using terminal multiplexers tmux(1) or screen(1) below. In some environments it might be necessary to also use pagsh(1), especially with kerberos, see below. Or nohup(1) might be of use.

Reconnect and restore a session[edit]

Sessions can be restored after either an intentional or accidental break using a terminal multiplexer like tmux(1) or screen(1). Here ssh(1) is to assume that the connection is broken after 5 seconds of not being able to reach the server and to exit.

# using screen
while ! ssh -t fred@example.org -o 'ServerAliveInterval 5' screen -d -RR;  
do true;
done

Each time ssh(1) exits, the shell tries to run it again and to look for a screen session to attach to. If the tcp connection is broken, none of the applications or sessions running inside the terminal multiplexer stop.

# using tmux
while ! ssh -t fred@example.org -o 'ServerAliveInterval 5' 'tmux attach -d || tmux new-session';  
do true;
done

The above examples give only an overly simplistic demonstration where at their most basic they are useful to resume a shell where it was after the tcp connection was broken. Both tmux(1) or screen(1) are quite capable of much more and worth exploring for travelers and telecommuters.

Keeping authentication tickets for a remote process after disconnecting[edit]

Authentication credentials are often deleted upon logout and thus any remaining processes no longer have access to whatever the authentication tokens were used for. In such cases, it is necessary to first create a new credential cache sandbox to run an independent process in before disconnecting. Using pagsh(1) is one solution for some environments.

$ pagsh
$ /usr/local/bin/a-slow-script.sh

Kerberos and AFS are two examples of services that require valid, active tickets.

Sharing a remote shell[edit]

Teaching, team programming, supervision and creating documentation are some examples of when it can be useful for two people to be sharing a shell. There are several options for read-only viewing as well as for multiple parties being able to read and write.

Read-only Monitoring or Logging[edit]

Pipes and redirects are a quick way to save output from an ssh session or to allow additional users to follow along read-only.

Sample use-case: Router needs to be reconfigured and is only available via serial console. Say your router is down and a consultant must log in via your laptop's 3G modem to access the router's serial console and you need to see what is done or help at certain stages. It is also very useful in documenting various activities, including configuration or installation.

It is also possible for advanced users of tmux(1) or screen(1) to allow read-only observers.

Read-only using tee(1)[edit]

Capture shell activity to a log file and optionally use tail to watch it real time. The utility tee(1), like a t-joint in plumbing, is used here to send output two directions. Output is sent with a pipe ( | ) to tee(1) which then sends the output both to stdout and to a file.

ssh fred@server.example.org | tee /tmp/session.log

The resulting file can be monitored live in another terminal using tail(1) or after the fact with a pager like less(1).

Force Serial Session with Logging using tee(1)[edit]

tee(1) can capture output from any program that can write to stdout. It is very useful for walking someone at a remote site through a process or supervising.

This example uses chroot(8) to keep the choice of actions as simple as possible. The guest user is a member of the group cconsult. The serial connection for my test is on device ttyUSB0, which is a USB to serial converter and cu(1) is used for the connection. tee(1) takes the output from cu(1) and saves a copy to a file for logging while the program is used.

# from sshd_config
Match Group cconsult
   ChrootDirectory /var/chroot-test
   AllowTCPForwarding no
   X11Forwarding no
   ForceCommand cu -s 19200 -l /dev/ttyUSB0 | tee /var/tmp/cu.log

Then one or more people can follow the activity in cu(1) as it happens, use tail.

# using the same logfile name as set in sshd_config
tail -f /var/tmp/cu.log.

Or the log can be edited and used for documentation.

Read-only using tmux(1) or screen(1) or tmux(1) is done by wrapping the (re-)connection attempt in a script.

#!/bin/sh

# try attaching to an existing screen session or,
# or if none exist, make a new screen session

/usr/bin/screen -x || \
  /usr/bin/screen \
     sh -c "/usr/bin/cu -s 19200 -l /dev/ttyUSB0 | \
     /usr/bin/tee /tmp/consultant.log"

Interactive Sharing Using a Terminal Multiplexer[edit]

The terminal multiplexers tmux(1) or screen(1) or tmux(1) can attach two or more users to the same session.[1] The session can either be read-only for some or read-write for all participants.

screen(1)[edit]

If the same account is going to share a session, then it's an easy procedure. In the one terminal, where sessionname is the name of your screen session, start a new session:

screen -S sessionname

In the other terminal, attach to that session:

screen -x sessionname

If two different accounts are going to share the same session, then the following steps are necessary. user 1 does this:

screen -S sessionname
^A :multiuser on
^A :acladd user2

Then user 2 does this:

screen -x user1/sessionname


In tmux(1) or screen(1), if more than one user account is used the aclchg command can remove write access for the other user: ^A :aclchg user -w "#". Note that tmux(1) or screen(1) must run as SUID for multiuser support. If it is not set, you will get an error message reminding you when trying to connect the second user. You might also have to set permissions for /var/run/screen to 755.

tmux(1)[edit]

If the same account is going to be sharing the session, then it's rather easy. In the first terminal, start tmux(1) where 'sessionname' is the session name:

tmux new-session -s sessionname

Then in the second terminal:

tmux attach-session -t sessionname


For different users, you have to set the permissions on the tmux(1) socket so that both users can read and write it. That will require a group which has both users as members.

In the first terminal, start tmux(1) where 'sessionname' is the session name and 'shareds' is the name of the socket:

tmux -S /tmp/shareddir/shareds new-session -s sessionname

Then chgrp(1) the socket (/tmp/shareddir/shareds) and the socket's directory to a group that both users share in common.

In the second terminal attach using that socket and session

tmux -S /tmp/shareddir/shareds attach-session -t sessionname

scripting[edit]

It is possible to automate some of the connection. Make a script, such as /usr/local/bin/screeners, then use that script with the ForceCommand directive. Here is an example of a script that tries to reconnect to an existing session. If no sessions already exist, then a new one is created and automatically establishes a connection to a serial device.

#!/bin/sh

# try attaching to an existing screen session,
# or if none exist, make a new screen session

/usr/bin/screen -x || \
  /usr/bin/screen \
     /bin/sh -c "/usr/bin/cu -s 19200 -l /dev/ttyUSB0 | \
     /usr/bin/tee /tmp/consultant.log"

Run a local process and capture remote data[edit]

Data can be produced on one system and used on another. This is different than tunnelling X, where both the program and the data reside on the other machine and only the graphical interface is displayed locally. The simplest way to read data on one machine and use it on another is to use I/O redirection.

ssh fred@server.example.org 'cat /etc/ntpd.conf' | diff /etc/ntpd.conf - 

Or

cat /etc/ntpd.conf | ssh fred@server.example.org 'diff /etc/ntpd.conf -' 

In the case where the local program expects to read a file from the remote machine, a named pipe can be used in conjunction with a redirect to transfer the data. In the following example, a named pipe is created as a transfer point for the data. Then ssh(1) is used to launch a remote process which sends output to stdout which is captured by a redirect on the local machine and sent to the named pipe. Any sudo(8) privileges for tcpdump(8) also need to operate without an interactive password, so great care and precision must be exercised to spell out in /etc/sudoers exactly which program and parameters are to be permitted and nothing more. The authentication for ssh(1) must also occur non-interactively, such as with a key and key agent. Once the configurations are set, ssh(1) is run and sent to the background after connecting. With ssh(1) in the background the local application is launched, in this case wireshark(1), a graphical network analyser, which is set to read the named pipe as input.

mkfifo -m 600 netdata

ssh -fq -i /home/fred/.ssh/key_rsa \
    'sudo tcpdump -lqi eth0 -w - "not port 22"' > netdata

wireshark -k -i netdata &

In this particular example, it is important to add a filter rule to tcpdump(8) itself to prevent an infinite feedback loop if ssh(1) is connecting over the same interface as the data being collected. This loop is prevented by excluding either the SSH port, the host used by the SSH connection, or the corresponding network interface.

On some systems, process substitution can be used to simplify the transfer of data between the two machines. Doing process substitution requires only a single line.

wireshark -k -i <( ssh -fq -i /home/fred/.ssh/key_rsa \
                  'sudo tcpdump -lqi eth0 -w - "not port 22"' )

However, process substitution is not POSIX compliant and thus not portable across platforms. It is limited to bash(1) only and not present in other shells. For portability, use a named pipe.

X11 Forwarding[edit]

It is possible to run graphical programs on the remote machine and have them display locally by forwarding X11, the current implementation of the X Window system. It is used to provide the graphical interface on many systems. See the website www.X.org for its history and technical details. X11 is built into most desktop operating systems. It is even distributed as part of Macintosh OS X, though there it is not the default method of graphical display.

X11 forwarding OpenSSH is off by default and must be enabled on both the ssh client and the server if it is to be used.

X11 uses a client server architecture with the X server being the part that does the display for the end user and the various programs acting as the clients connecting to the server. By putting the client and server on different machines and forwarding the X11 connections, it is possible to run programs on other computers but have them displayed and available as if they were on the user's computer.

A note of caution is warranted. Allowing the remote machine to forward X11 connections will allow it and its applications to access many devices and resources on the machine hosting the X server. Regardless of the intent of the users, these are the devices and other resources accessible to the user account. So forwarding should only be done when the other machine, that user account account and its applications are reliable.

On the server side, to enable X11 forwarding by default, put the line below in sshd_config(5):

X11Forwarding yes

On the client side, forwarding of X11 is also off by default, but can be enabled using three different ways. It can be enabled using -X or -Y as a run-time argument or in ssh_config(5).

ssh -l fred -X desk.example.org 

Using ssh_config to specify X11 forwarding[edit]

Or X11 forwarding can be enabled in /etc/ssh_config for all accounts. In ~/.ssh/config and /etc/ssh/ssh_config it can be enabled for all hosts or for specific ones.

#  Enable for all hosts.  Maybe this is not a good idea.
X11Forwarding yes

It is possible to use ssh_config or ~/.ssh/config to limit forwarding by default to an individual host by hostname or ip number.

# enable for one machine by name
Host desk.example.org 
        X11Forwarding yes

# enable for one machine by ip number
Host 192.168.111.25 
        X11Forwarding yes

Or likewise use limited pattern matching to allow forwarding for a subdomain or a range of ip addresses:

# Any host in the subdomain
Host *.pool.example.org 
        X11Forwarding yes

# Any host in the range 192.168.100.100 - 192.168.100.109
Host 192.168.100.10?
        X11Forwarding yes

# Any host in the range 192.168.123.1 - 192.168.123.255
Host 192.168.123.*
        X11Forwarding yes

X11 is built-in to most desktop systems with optional add-on for OS X which has its roots in NextStep. X11 support may be missing from some particularly outdated, legacy platforms. But even there it is often possible to retrofit them, one example being the tool Xming.