OpenSSH/Cookbook/File Transfer with SFTP

 

Basic SFTP service requires no additional setup. It is a built-in part of the OpenSSH server and it is the subsystem sftp-server(8) which then implements an SFTP file transfer. See the manual page for sftp-server(8). Alternately, the internal-sftp subsystem can implement an in-process SFTP server which may simplify configurations using ChrootDirectory to force a different filesystem root on clients.

On the client, the same options and tricks are available for SFTP as for the regular SSH clients. However, if using an odd port to connect, note that the SFTP client uses -P and not -p for that. Also, some client options may have to be specified with the full SSH client option name using an -o argument. For many dedicated graphical SFTP clients, it is possible to use a regular URL to point to the target, specifying sftp:// as the protocol. Many file managers nowadays have built-in support for SFTP and could accept a URL like sftp://fred@server.example.org/home/fred/Documents/. See the earlier section GUI Clients about that.

Basic SFTP

SFTP provides a very easy to use and very easy to configure option for accessing a remote system. Just to say it again, regular SFTP access requires no additional changes from the default configuration. The usual clients can be used or special ones like sshfs(1).

Automated SFTP

SFTP uploads or downloads can be automated. The prerequisite is key-based authentication. Once key-based authentication is working, a batch file can be used to carry out activities via SFTP. See the batchfile option -b in sftp(1) for details.

$ sftp -b /home/fred/cmds.batch -i /home/fred/.ssh/server_key_ed25519 server.example.org:/home/fred/logs/

If a dash (-) is used as the batch file name, SFTP commands will be read from stdin.

$ echo "put /var/log/foobar.log" | sftp -b - -i /home/fred/.ssh/server_key_ed25519 server.example.org:/home/fred/logs/

More than one SFTP command can be sent, but then it is better to use the batch file mode.

$ echo -e "put /var/log/foobar.log\nput /var/log/munged.log" | sftp -b - -i /home/fred/.ssh/server_key_ed25519 server.example.org:/home/fred/logs/

The batch file mode can be very useful in cron jobs and in scripting.

SFTP-only Accounts

It is possible to limit members of a specific group to using only SFTP for interaction with the server using the ForceCommand directive in sshd_config(5). In that way, interactive shells and arbitrary commands become prohibited.

Subsystem sftp internal-sftp

Match Group sftp-only
	AllowTCPForwarding no
	X11Forwarding no
	ForceCommand internal-sftp

In that example, accounts to be restricted to SFTP must be added to the group 'sftp-only' first. Once in that group, the Match block applies the restrictions upon login. Note that disabling TCP forwarding does not improve security unless users are also denied shell access using the ForceCommand, as they can otherwise in principle install their own forwarders.

See PATTERNS in ssh_config(5) for more information on patterns available to Match.

Three Ways of Setting Permissions for Chrooted SFTP-Only Accounts

It is common for a group of accounts to need to read and write files to their home directories on the server while having little or no reason to access the rest of the file system. The chroot(8) utility can do that and OpenSSH provides a very easy to use and very easy to configure chroot for its internal SFTP subsystem.

In some cases, it is enough to chroot users to their home directories. However, this may not be as straight forward as it seems because in most cases home directories aren't owned by root and allow writing by at least one user. SFTP chroot requires that the chroot target directory and all parent directories are owned by root and not writable by any others. This can cause some difficulty but is quite necessary. Without such ownership restrictions, it is quite feasible to escape the chroot.

There are at least three ways set permissions for home directories for chrooted SFTP-only accounts. They each have strong advantages and a few drawbacks. Therefore each will suit some particular situations but not others.

Option 1: Automatically Setting Only the Starting Directory

If it is not practical to have the various account's home directories owned by root, a compromise can be made. The ChrootDirectory directive can point to /home, which must be owned by root anyway, and then a ForceCommand can then designate the user's home directory as the starting directory using the -d option.

Match Group sftp-only
        ChrootDirectory /home/
        ForceCommand internal-sftp -d %u

There the SFTP session chroots to /home/ and then the internal SFTP server immediately changes the directory to the user directory below it. The file permissions to go with those settings could look something like the following:

$ ls -lhd /home/ /home/*/
drwxr-xr-x  4 root  root  4.0K Aug  4 20:47 /home/
drwxr-x--- 14 user1 user1 4.0K Aug  4 20:47 /home/user1/
drwxr-x--- 14 user2 user2 4.0K Aug  4 20:47 /home/user2/

However, the SFTP client could still change directories and navigate to any of the other directories under /home/ the usual way. If it is necessary to hide the contents of the other home directories, chmod(1) can be used to eliminate read access to the chroot target, but leave the eXecutable bit set so that the directory can still be traversed without seeing the contents. Thus permissions could then be 0111 for /home with 0750, 0700, 0770, or 2770, and so on for the actual home directories underneath it. Be sure to pay attention to the group memberships as well.

Option 2: Nested Directories

Nesting a directory under the chroot is one more way to hide all the other directories found within the chroot target. The destination subdirectory would usually be writable by the unprivileged account, but of course the chroot target above it cannot not be writable by any other account than root.

A common use case is when a web server's document root or server root is the chroot target. For example, if each site has its own hierarchy under /var/www/ such as /var/www/site1/ then chroot can be put to use as follows:

Match Group team1
	ChrootDirectory /var/www/
	ForceCommand internal-sftp -d site1

Match Group team2
	ChrootDirectory /var/www/
	ForceCommand internal-sftp -d site2

The site directories can then be group writable while the parent /var/www/ remains read-only for non-root users.

A similar approach can be taken with home directories. In this example, the ChrootDirectory directive lock the accounts within the group 'sftp-only' into the directory above their respective homes and that is followed by the ForceCommand directive which then launches the session inside the actual home directory using the -d option.

Match Group sftp-only
        ChrootDirectory /home/%u
        AllowTCPForwarding no
        X11Forwarding no
        ForceCommand internal-sftp -d %u

That way, with the right directory permission settings, and once logged in, the accounts in that group can only ever see their own files. The %u is there twice on purpose, once as the chroot target and then again for the destination with the same name underneath. Note the directory names with their ownership and permissions for the directories to go with those settings:

$ ls -lhd /home/ /home/*/ /home/*/*/
drwxr-xr-x  4 root  root  4.0K Aug  4 20:47 /home/
drwxr-x---  3 root  user1 4.0K Aug  4 20:47 /home/user1/
drwxr-x---  3 root  user2 4.0K Aug  4 20:47 /home/user2/
drwxr-x--- 14 user1 user1 4.0K Aug  4 20:47 /home/user1/user1/
drwxr-x--- 14 user2 user2 4.0K Aug  4 20:47 /home/user2/user2/

This particular arrangement also makes it easier to add chrooted, interactive shell access later on because system directories can be added to the chroot without being accessible by any of the other accounts.

Option 3: Split Ownership of the Home Directory and Its Contents

As seen above there are several ways to deal with a chrooted SFTP service. A third way would be to set the directory to be owned by root but not writable by the unprivileged account(s) but manually create the next layers of files and directories. In this approach, the chroot target is still only writable by root, but it has been populated in advance with all the necessary files and directories, which are then set manually to have the appropriate permissions for use by the unprivileged account.

Match Group team3
	ChrootDirectory /home/%u
	ForceCommand internal-sftp

Then with the following permissions, as a member of the group 'team3' the account 'fred' can log in and use any of the pre-made subdirectories and files in the usual ways, but cannot add anything to the chroot target itself.

$ ls -lhd /home/ /home/fred/ /home/fred/*
drwxr-xr-x  68 root root 4.0K Sep 4 15:40 /home/
drwxr-xr-x  21 root fred 4.0K Sep 4 15:41 /home/fred/
drwxr-xr-x   8 fred fred 4.0K Sep 4 15:44 /home/fred/Documents
drwxr-xr-x   9 fred fred 4.0K Sep 4 15:41 /home/fred/Music
drwxr-xr-x 145 fred fred 4.0K Sep 4 15:41 /home/fred/Pictures
drwxr-xr-x   5 fred fred 4.0K Sep 4 15:41 /home/fred/Videos
drwxr-xr-x  98 fred fred 4.0K Sep 4 15:41 /home/fred/WWW
-rw-------   1 fred fred     19407 Sep  4 15:42 .bash_history
-rw-r-----   1 fred fred       220 Sep  4 15:42 .bash_logout
-rw-r-----   1 fred fred      1468 Sep  4 15:42 .bashrc
-rw-------   1 fred fred     19407 Sep  4 15:42 .tmux.conf

This method is quick to set up but the drawback is that it requires system administrator intervention each and every time a new directory or a new file is to be added to the home directory. Root permissions are needed to add anything and then set the permissions (and perhaps ownership) to allow the the unprivileged account write access without disturbing the settings for the chroot target itself.

Umask

Starting with OpenSSH 5.4, the sftp-server(8) can set a umask to override the default one set by the user’s account. The in-process SFTP server, internal-sftp, accepts the same options as the external SFTP subsystem.

Subsystem sftp internal-sftp -u 0022

However it is important to remember that umasks only restrict permissions, never loosen them.

Earlier versions of the SFTP servers can do the same thing through the use of a helper script, but that complicates chrooted directories a lot. The helper script can be a regular script or it can be embedded inline in the configuration file though neither works easily in a chroot jail. It’s often easier to get a newer version of sshd(8) which supports umask as part of the server’s configuration. Here is an inline helper script for umask in OpenSSH 5.3 and earlier, based on one by gilles@

Subsystem sftp /bin/sh -c 'umask 0022; /usr/libexec/openssh/sftp-server'

Either way, this umask is server-side only. The original file permissions on the client side will usually, but not always, be used when calculating the final file permissions on the server. This depends on the client itself. Most clients pass the file permissions on to the server, FileZilla being a notable exception. As such, permissions can generally be tightened but not loosened. For example, a file that is mode 600 on the client will not be automatically become 664 or anything else less than the original 600 regardless of the server-side umask. That is unless the client does not forward the permissions, in which case only the server's umask will be used. So for most clients, if you want looser permissions on the uploaded file, change them on the client side before uploading.

Further Restricting Chrooted SFTP

There are several additional common scenarios where chrooted access is restricted further.

Chrooted SFTP to Shared Directories

Another common use case is to chroot a group of users to different levels of the web server they are responsible for. For obvious reasons, symbolic links going from inside the jail to parts of the filesystem outside the chroot jail are not accessible to the chrooted users. So directory hierarchies must be planned more carefully if there are special combinations of access. See "Option 2: Nested Directories in" the earlier section on "Chrooted SFTP-Only Accounts".

In these kinds of directories, it may be useful to give different levels of access to more than just one group. In that case, ACLs might be needed.

Chrooted SFTP Accounts Accessible Only from Particular Addresses

More complex matching can be done to make it possible to allow a group of users to use SFTP, and not a shell login, but only when they log in from a specific address or range of addresses. They get no access at all when trying to connect from elsewhere but if they log in from the right addresses, they get SFTP and only SFTP, no shell. All Match conditions, both the affirmative and negative matches, need to be accounted for.

Subsystem sftp internal-sftp

Match Group sftp-only, Address 192.0.2.10
        AllowTCPForwarding no
        X11Forwarding no
        ForceCommand internal-sftp
        ChrootDirectory  /home/servers/

Match Group sftp-only, Address *,!192.0.2.10
        DenyGroups sftp-only

Note that for negation a wildcard must be specified first and then the address or range to be excluded following it. Mind the spaces or lack thereof. Similar matching can be done for a range of addresses by specifying the addresses in CIDR address/mask format, such as 192.0.2.0/24. Any number of criteria can be specified and only if all of them are met then the directive in the subsequent lines within the Match block take effect.

The first Match block which fits is the one that takes effect, so care must be taken when constructing conditional blocks to make them fit the precise situation desired. Also, any situations that don't fit a Match conditional block may fall through the cracks and will get the general configuration settings whatever they may be. Specific user and source address combinations can be tested with the configurations using the -T and -C options with the server for more options. See the section Debugging a Server Configuration for more.

Chrooted SFTP with Logging

Chroots are more difficult to set up if the internal-sftp in-process SFTP server is not used. In that case, the logging daemon must have a socket in the chroot directory for the sftp-server(8) subsystem to access, usually with the name ./dev/log and then syslog(8) must be configured to read that new socket in addition to what it is already set to read. Either way, see the manual page for the sftp-server(8) for more as the both the external and internal subsytems accept the same options. Then see the chapters on Logging for specifics.

Chrooted Login Shells

Making a chroot jail for interactive shells is difficult. The chroot and all its components must be root-owned directories that are not writable by any other user or group. The ChrootDirectory must contain the necessary files and directories to support the user's session. For an interactive session this requires at least a shell, typically bash(1), ksh(1), or sh(1), and basic device nodes inside /dev such as null(4), zero(4), stdin(4), stdout(4), stderr(4), arandom(4), and tty(4) devices. Paths may contain the following tokens that are expanded at runtime once the connecting user has been authenticated: %% is replaced by a literal '%', %h is replaced by the home directory of the user being authenticated, and %u is replaced by the username of that user.

sshfs(1) — SFTP File Transfer Via Local Folders

Another way to transfer files back and forth via SFTP, or even use the files remotely, is to use sshfs(1), the SSH userspace file system. It is a Filesystem in Userspace (FUSE) client operating over SFTP utilizing the server's SFTP-subsystem. It can make a directory on the remote server accessible as if it were a local directory and in that way the files and directories within the mount point can be accessed by any program. The user must have read-write privileges for the local mount point to make use of sshfs(1).

The following creates a mount point, named mountpoint here, in the home directory if none exists. Then sshfs(1) mounts the remote server.

$ test -d ~/mountpoint || mkdir --mode 700 ~/mountpoint
$ sshfs fred@server.example.org:. ~/mountpoint

Reading or writing files via the SSHFS mount point is actually using SFTP to transfer data to or from the remote system. The amount of bandwidth consumed by the transfers can usually be reduced using compression. That can be important if the network connection has bandwidth caps or per-unit fees. However, if speed is the only consideration, then compression can be avoided as it makes the transfer slower when the processors on either end are busy or not powerful enough or both. About the only way to know is to test and see which method is faster. Below, compression is specified with -C.

$ sshfs -C fred@server.example.org:. ~/mountpoint

Or mount the remote directory with debugging output:

$ sshfs -o sshfs_debug fred@server.example.org:. /home/fred/mountpoint

Named pipes will not work over sshfs(1).

To unmount these remote directories and close the SFTP session use fusermount -u if in the interactive shell. Though on some systems the regular umount(8) utility will work just fine instead. Or use the graphical file manager's eject/unmount feature if in a graphical session on a desktop system.

Using sshfs(1) With A Key

The ssh_command option is used to pass parameters from the SSHFS client onward to ssh(1) underneath it. In this example it is used to have ssh(1) point to a specific key to be used for authentication when mounting the remote directory. The remote directory here is /usr/src, and the local directory is /home/fred/src.

$ sshfs -o ssh_command="ssh -i /home/fred/.ssh/server_key_ed25519" fred@server.example.org:/usr/src /home/fred/src/

If a usable key is already loaded into an available SSH agent, then ssh(1) should find it automatically and use it on behalf of sshfs(1) without needing intervention.