OpenSSH/Cookbook/Remote Processes

From Wikibooks, open books for an open world
Jump to: navigation, search


One of the main functions of OpenSSH is that of accessing and running programs on other systems. In addition to an interactive login, ssh(1) can be used to simply execute a program or script. Logout is automatic when the program or script has run its course. Some combinations are readily obvious. Others require more careful planning. Sometimes it is enough of a clue just to know that something can be done, at other times more detail is required. A number of examples of useful combinations of using OpenSSH to run remote tasks follow.

Run a Remote Process[edit]

An obvious use of ssh(1) is to use the shell interactively. That is, after all, one of the main purposes of the program. There are several ways to expand upon that, either interactively or as part of unattended scripts. ssh(1) returns whatever exit value was returned by the remote process. When a remote process is completed, ssh(1) will terminate and pass on the exit value of the last remote process to complete. So in this way, it can be used in scripts and the outcome of the remote processes can be used.

The following will return success, 0, on the local system where ssh(1) was run.

$ ssh -l fred /bin/true
$ echo $?

The following will return failure, 1, on the local system where ssh(1) was run.

$ ssh -l fred /bin/false
$ echo $?

If any other values, from 0 to 255, were returned, ssh(1) will pass them on from the remote host.

Run a Remote Process and Capture Output Locally[edit]

Output from programs run on the remote machine can be saved locally using a normal redirect. Here we run dmesg(8) on the remote machine:

$ ssh -l fred dmesg > dmesg.from.server.log

Interactive processes will be difficult or impossible to operate in that manner because no output will be seen. For interactive processes requiring any user input, output can be piped through tee instead to send the output both to the file and to stdout. This example runs an anonymous FTP session remotely and logs the output locally.

$ ssh -l fred "ftp -a anotherserver" | tee ftp.log

It may be necessary to force pseudo-TTY allocation to get both input and output to be properly visible.

$ ssh -t -l fred "ftp -a anotherserver" | tee /home/fred/ftp.log

The simplest way to read data on one machine and process it on another is to use pipes.

$ ssh 'cat /etc/ntpd.conf' | diff /etc/ntpd.conf - 

Run a Local Process and Capture Remote Data[edit]

Data can be produced on one system and used on another. This is different than tunnelling X, where both the program and the data reside on the other machine and only the graphical interface is displayed locally. Again, the simplest way to read data on one machine and use it on another is to use pipes.

$ cat /etc/ntpd.conf | ssh 'diff /etc/ntpd.conf -' 

In the case where the local program expects to read a file from the remote machine, a named pipe can be used in conjunction with a redirect to transfer the data. In the following example, a named pipe is created as a transfer point for the data. Then ssh(1) is used to launch a remote process which sends output to stdout which is captured by a redirect on the local machine and sent to the named pipe. Any sudo(8) privileges for tcpdump(8) also need to operate without an interactive password, so great care and precision must be exercised to spell out in /etc/sudoers exactly which program and parameters are to be permitted and nothing more. The authentication for ssh(1) must also occur non-interactively, such as with a key and key agent. Once the configurations are set, ssh(1) is run and sent to the background after connecting. With ssh(1) in the background the local application is launched, in this case wireshark(1), a graphical network analyser, which is set to read the named pipe as input.

$ mkfifo -m 600 netdata

$ ssh -fq -i /home/fred/.ssh/key_rsa \
        'sudo tcpdump -lqi eth0 -w - "not port 22"' > netdata

$ wireshark -k -i netdata &

In this particular example, it is important to add a filter rule to tcpdump(8) itself to prevent an infinite feedback loop if ssh(1) is connecting over the same interface as the data being collected. This loop is prevented by excluding either the SSH port, the host used by the SSH connection, or the corresponding network interface.

On some systems, process substitution can be used to simplify the transfer of data between the two machines. Doing process substitution requires only a single line.

$ wireshark -k -i <( ssh -fq -i /home/fred/.ssh/key_rsa \
        'sudo tcpdump -lqi eth0 -w - "not port 22"' )

However, process substitution is not POSIX compliant and thus not portable across platforms. It is limited to bash(1) only and not present in other shells. For portability, use a named pipe.

Run a Remote Process While Either Connected or Disconnected[edit]

There are several different ways to leave a process running on the remote machine. If the intent is to come back to the process and check on it periodically then a terminal multiplexer is probably the best choice. For simpler needs there are other approaches.

Run a Remote Process in the Background While Disconnected[edit]

Many routine tasks can be set in motion and then left to complete on their own without needing to stay logged in. When running remote process in background it is useful to spawn a shell just for that task.

$ ssh -t -l fred 'sh -c "tar zcf /backup/usr.tgz /usr/" &' 

Another way is to use a terminal multiplexer, like tmux(1) or screen(1). An advantage with them is being able to reconnect and follow the progress from time to time.

Here screen(1) is launched remotely. The two options together mean to create a new session if none exist, or re-attach to the session if one is already runing. So if no screen(1) session is running, one will be created.

$ ssh -t -l fred "screen -d -R"

Once a screen(1) session is running, it is possible to detach it and close the SSH connection without disturbing the background processes it may be running. That can be particularly useful when hosting certain game servers on a remote machine. Then the terminal session can then be reattached in progress with the same two options.

$ ssh -t -l fred "screen -d -R"  

More on using terminal multiplexers tmux(1) or screen(1) below. In some environments it might be necessary to also use pagsh(1), especially with kerberos, see below. Or nohup(1) might be of use.

Keeping Authentication Tickets for a Remote Process After Disconnecting[edit]

Authentication credentials are often deleted upon logout and thus any remaining processes no longer have access to whatever the authentication tokens were used for. In such cases, it is necessary to first create a new credential cache sandbox to run an independent process in before disconnecting. Using pagsh(1) is one solution for some environments.

$ pagsh
$ /usr/local/bin/

Kerberos and AFS are two examples of services that require valid, active tickets.

Reconnect and Restore a Session[edit]

Running sessions can be restored after either an intentional or accidental break using a terminal multiplexer like tmux(1) or screen(1). Here ssh(1) is to assume that the connection is broken after 15 seconds (three tries of five seconds each) of not being able to reach the server and to exit.

$ while ! ssh -t -o 'ServerAliveInterval 5' 'tmux attach -d || tmux new-session';  
$ do true;
$ done

Each time ssh(1) exits, the shell tries to run it again and to look for a tmux session to attach to. If the TCP connection is broken, none of the applications or sessions running inside the terminal multiplexer stop.

$ while ! ssh -t -o 'ServerAliveInterval 5' screen -d -R;  
$ do true;
$ done

The above examples give only an overly simplistic demonstration where at their most basic they are useful to resume a shell where it was after the TCP connection was broken. Both tmux(1) and screen(1) are quite capable of much more and worth exploring for travelers and telecommuters.

Sharing a Remote Shell[edit]

Teaching, team programming, supervision and creating documentation are some examples of when it can be useful for two people to be sharing a shell. There are several options for read-only viewing as well as for multiple parties being able to read and write.

Read-only Monitoring or Logging[edit]

Pipes and redirects are a quick way to save output from an SSH session or to allow additional users to follow along read-only.

One sample use-case is when a router needs to be reconfigured and is only available via serial console. Say the router is down and a consultant must log in via another user's laptop's connection to access the router's serial console and it is necessary to supervise what is done or help at certain stages. It is also very useful in documenting various activities, including configuration or installation.

It is also possible for advanced users of tmux(1) or screen(1) to allow read-only observers.

Read-only Using tee(1)[edit]

Capture shell activity to a log file and optionally use tail to watch it real time. The utility tee(1), like a t-joint in plumbing, is used here to send output two directions. Output is sent with a pipe ( | ) to tee(1) which then sends the output both to stdout and to a file.

$ ssh | tee /tmp/session.log

The resulting file can be monitored live in another terminal using tail(1) or after the fact with a pager like less(1).

Force Serial Session with Remote Logging Using tee(1)[edit]

tee(1) can capture output from any program that can write to stdout. It is very useful for walking someone at a remote site through a process or supervising.

This example uses chroot(8) to keep the choice of actions as simple as possible. The guest user is a member of the group 'cconsult'. The serial connection for my test is on device ttyUSB0, which is a USB to serial converter and cu(1) is used for the connection. tee(1) takes the output from cu(1) and saves a copy to a file for logging while the program is used. The following is from sshd_config(5)

Match Group cconsult
        ChrootDirectory /var/chroot-test
        AllowTCPForwarding no
        X11Forwarding no
        ForceCommand cu -s 19200 -l /dev/ttyUSB0 | tee /var/tmp/cu.log

Then one or more people can follow the activity in cu(1) as it happens using tail(1) by pointing it at the log file.

$ tail -f /var/tmp/cu.log

Or the log can be edited and used for documentation.

Read-only using tmux(1) or screen(1) is done by wrapping the (re-)connection attempt in a script.


# try attaching to an existing screen session or,
# or if none exist, make a new screen session

/usr/bin/screen -x || \
        /usr/bin/screen \
            sh -c "/usr/bin/cu -s 19200 -l /dev/ttyUSB0 | \
            /usr/bin/tee /tmp/consultant.log"

It is possible to automate some of the connection. Make a script, such as /usr/local/bin/screeners, then use that script with the ForceCommand directive. Here is an example of a script that tries to reconnect to an existing session. If no sessions already exist, then a new one is created and automatically establishes a connection to a serial device.


# try attaching to an existing screen session,
# or if none exist, make a new screen session

/usr/bin/screen -x || \
        /usr/bin/screen \
            /bin/sh -c "/usr/bin/cu -s 19200 -l /dev/ttyUSB0 | \
            /usr/bin/tee /tmp/consultant.log"

Interactive Sharing Using a Terminal Multiplexer[edit]

The terminal multiplexers tmux(1) or screen(1) can attach two or more people to the same session.[1] The session can either be read-only for some or read-write for all participants.


If the same account is going to be sharing the session, then it's rather easy. In the first terminal, start tmux(1) where 'sessionname' is the session name:

$ tmux new-session -s sessionname

Then in the second terminal:

$ tmux attach-session -t sessionname

For different users, you have to set the permissions on the tmux(1) socket so that both users can read and write it. That will require a group which has both users as members.

In the first terminal, start tmux(1) where 'sessionname' is the session name and 'shareds' is the name of the socket:

$ tmux -S /tmp/shareddir/shareds new-session -s sessionname

Change the group of the socket (/tmp/shareddir/shareds) and the socket's directory to a group that both users share in common. Then make sure that the socket permissions allow the group to write the socket.

$ chgrp foo /tmp/shareddir/
$ chgrp foo /tmp/shareddir/shareds
$ chmod g+rx /tmp/shareddir/
$ chmod g+rw /tmp/shareddir/shareds

Then in the second terminal attach using that socket and session

$ tmux -S /tmp/shareddir/shareds attach-session -t sessionname

If the same account is going to share a session, then it's an easy procedure. In the one terminal, where 'sessionname' is the name of your screen session, start a new session:

$ screen -S sessionname

In the other terminal, attach to that session:

$ screen -x sessionname

If two different accounts are going to share the same session, then the following steps are necessary. 'user1' does this:

$ screen -S sessionname
^A :multiuser on
^A :acladd user2

Then 'user2' does this:

$ screen -x user1/sessionname

In tmux(1) or screen(1), if more than one user account is used the aclchg command can remove write access for the other user: ^A :aclchg user -w "#". Note that tmux(1) or screen(1) must run as SUID for multiuser support. If it is not set, you will get an error message reminding you when trying to connect the second user. You might also have to set permissions for /var/run/screen to 755.

Display Remote Graphical Programs Locally Using X11 Forwarding[edit]

It is possible to run graphical programs on the remote machine and have them display locally by forwarding X11, the current implementation of the X Window system. It is used to provide the graphical interface on many systems. See the website for its history and technical details. X11 is built into most desktop operating systems. It is even distributed as part of Macintosh OS X, though there it is not the default method of graphical display.

X11 forwarding OpenSSH is off by default and must be enabled on both the SSH client and the server if it is to be used.

X11 uses a client server architecture with the X server being the part that does the display for the end user and the various programs acting as the clients connecting to the server. By putting the client and server on different machines and forwarding the X11 connections, it is possible to run programs on other computers but have them displayed and available as if they were on the user's computer.

A note of caution is warranted. Allowing the remote machine to forward X11 connections will allow it and its applications to access many devices and resources on the machine hosting the X server. Regardless of the intent of the users, these are the devices and other resources accessible to the user account. So forwarding should only be done when the other machine, that user account account and its applications are reliable.

On the server side, to enable X11 forwarding by default, put the line below in sshd_config(5):

X11Forwarding yes

On the client side, forwarding of X11 is also off by default, but can be enabled using three different ways. It can be enabled using -X or -Y as a run-time argument or in ssh_config(5).

$ ssh -l fred -X 

The connection will be slow, however. If responsiveness is a factor, it may be relevant to consider a SOCKS proxy instead or some other technology all together like FreeNX.

Using ssh_config to specify X11 forwarding[edit]

Or X11 forwarding can be enabled in /etc/ssh_config for all accounts. In ~/.ssh/config and /etc/ssh/ssh_config it can be enabled for all hosts or for specific ones.

#  Enable for all hosts.  Maybe this is not a good idea.
X11Forwarding yes

It is possible to use ssh_config(5) ~/.ssh/config to limit forwarding by default to an individual host by hostname or IP number.

# enable for one machine by name
        X11Forwarding yes

# enable for one machine by IP number
        X11Forwarding yes

Or likewise use limited pattern matching to allow forwarding for a subdomain or a range of IP addresses:

# Any host in the subdomain
Host * 
        X11Forwarding yes

# Any host in the range -
        X11Forwarding yes

# Any host in the range -
Host 192.168.123.*
        X11Forwarding yes

X11 is built-in to most desktop systems with optional add-on for OS X which has its roots in NextStep. X11 support may be missing from some particularly outdated, legacy platforms. But even there it is often possible to retrofit them, one example being the tool Xming.

Locking Down a Restricted Shell[edit]

A restricted shell sets up a more controlled environment than what is normally provided by a standard interactive shell. Though it behaves almost identically to a standard shell, it has many exceptions regarding capabilities that are disallowed or disabled. The restrictions inlude, but are not limited to, the following:

  • The SHELL, ENV, and PATH variables cannot be changed.
  • Programs can't be run with absolute or relative paths.
  • Redirections that create files can't be used (specifically >, >|, >>, <>).

Common high-end shells like bash(1), ksh(1), and zsh(1) all can be launched in restricted mode. See the manual pages for the individual shells for the specifics of how to invoke restrictions.

Even with said restrictions, there are several ways by which it is trivial to escape from a restricted shell: If normal shells are available anywhere in the path, they can be launched instead. If regular programs in the available path provide shell escapes to full shells, they too can be used. Finally, if sshd(8) is configured to allow arbitrary programs to be run indepently of the shell, a full shell can be launched instead. So there's more to safely using restricted shells than just setting the account's shell to /bin/rbash and calling it a day. Several steps are needed to make that as difficult as possible to escape the restrictions, especially over SSH.

(The following steps assume familiarity with the appropriate system administration tools and their use. Their selection and use are not covered here.)

First, a directory of a small handful symbolic links pointing to white-listed programs must be created and populated. The links point to the programs that the account should be able to run when the directory is added to their PATH environent variable. Any programs in these directories should be free of shell escapes and definitely not include shells themselves. Note that these programs might still allow exploration of the system at large, if not locked further into a chroot or jail. Symlinks are used because the originals are, hopefully, maintained by package management software and should not be moved. Symlinks are used instead of hard links if the original and the whitelisted directory exist on different partitions.

$ ls -l /usr/local/rbin/
total 8
lrwxr-xr-x  1 root  wheel    22 Jan 17 23:08 angband -> /usr/local/bin/angband
lrwxr-xr-x  1 root  wheel     9 Jan 17 23:08 date -> /bin/date
-rwxr-xr-x  1 root  wheel  2370 Jan 17 23:18 help
lrwxr-xr-x  1 root  wheel    12 Jan 17 23:07 man -> /usr/bin/man
lrwxr-xr-x  1 root  wheel    13 Jan 17 23:09 more -> /usr/bin/more
lrwxr-xr-x  1 root  wheel    28 Jan 17 23:09 nethack -> /usr/local/bin/nethack-3.4.3

Next, created a truncated user .profile for that account and chown both it and the home directory it is in to the user root. Allow the account's own group to read both the file and the directory.

$ cat /home/fred/.profile

$ ls -lh /home/fred/.profile
-rw-r--r--  1 root  fred     48B Jan 17 23:20 /home/fred/.profile

$ ls -lhd /home/fred/
drwxr-xr-x  3 root  fred    512B Jan 17 23:20 /home/fred/

Next, create a group for the locked down account(s) and populate it. Here the account is in the group games and will be restricted through membership in that group.

$ groups fred
fred games

Next, lock down SSH access through use of a ForceCommand directive in the server congiguration and apply it to the selected group. This is necessary to prevent trivial circumvention through the SSH client, such as with ssh -t /bin/sh or similar. Remember to disable forwarding if it is not needed. For example, the following can be appended to sshd_config(5) so that any account in the group games gets a restricted shell no matter what they try with the SSH client.

Match Group games
        X11Forwarding no
        AllowTcpForwarding no
        ForceCommand rksh -l

Note that the restricted shell is invoked with the -l option by the ForceCommand so that it will be a login shell that reads and executes the contents of /etc/profile and $HOME/.profile if they exist and are readable. This is necessary to set the custom PATH environment variable. Again, be sure that $HOME/.profile is not in any way editable or overwritable by the restricted account. Also note that this disables SFTP access by that account, which prevents quite a bit of additional mischief.

Last but not least, set the account's login shell to the restricted shell. Include the full path to the restricted shell. It might also be necessary to add it to the list of approved shells found in /etc/shells first.