OpenSSH/Cookbook/Load Balancing

 

MaxStartups

Random early drop can be enabled by specifying the three colon-separated values start:rate:full. After the number of unauthenticated connections reaches the value specified by start, sshd(8) will begin to refuse new connections at a percentage specified by rate. The proportional rate of refused connections then increases linearly as the limit specified by full is approached until 100% is reached. At that point all new attempts at connection are refused until the backlog goes down.

MaxStartups   10:30:100

For example, if MaxStartups 5:30:90 is given in sshd_config(5), then starting with 5 new connections pending authentication the server will start to drop 30% of the new connections. By the time the backlog increases to 90 pending unauthenticated connections, 100% will be dropped.

In the default settings, the value for full has been increased to 100 pending connections to make it harder to succumb to a denial of service due to attack or heavy load. So the new default is 10:30:100.

Alternately, if the number of incoming connections is not managed at the network level by a packet filter or other tricks like round-robin DNS, it is possible to limit it at the SSH server itself. Setting MaxStartups to an integer sets a hard limit to the maximum number of concurrent unauthenticated connections to the SSH daemon.

MaxStartups   10

Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for another connection. The old default was 10.

Preventing Timeouts Of A Not So Active Session

There are two connections that can be tracked during an SSH session, the network's TCP connection and the encrypted SSH session traveling on top of it. Some tunnels and VPNs might not be active all the time, so there is the risk that the session times out or even that the TCP session times out at the router or firewall. The network connection can be be tracked with TCPKeepAlive, but is not an accurate indication of the state of the actual SSH connections. It is, however, a useful indicator of the status of the actual network. Either the client or the server can counter that by keeping either the encrypted connection active using a heartbeat.

On the client side, if the global client configuration is not already set, individuals can use ServerAliveInterval to choose an interval in seconds for server alive heartbeats and use ServerAliveCountMax to set the corresponding maximum number of missed client messages allowed before the encrypted SSH session is considered closed.

ServerAliveInterval  15
ServerAliveCountMax  4

On the server side, ClientAliveInterval sets the interval in seconds between client alive heartbeats. ClientAliveCountMax sets the maximum number of missed client messages allowed before the encrypted SSH session is considered closed. If no other data has been sent or received during that time, sshd(8) will send a message through the encrypted channel to request a response from the client. If sshd_config(5) has ClientAliveInterval set to 15, and ClientAliveCountMax set to 4, unresponsive SSH clients will be disconnected after approximately 60 (= 15 x 4) seconds.

ClientAliveInterval  15
ClientAliveCountMax  4

If a time-based RekeyLimit is also used but the time limit is shorter than the ClientAliveInterval heartbeat, then the shorter re-key limit will be used for the heartbeat interval instead.

This is more or less the same principal as with the server. Again, that is set in ~/.ssh/config and can be applied globally to all connections from that account or selectively to specific connections using a Host or Match block.

Ensuring Timeouts Of An Inactive Interactive Session

If the server is no longer disconnecting idle SSH accounts when they reach the timeout configured by the ClientAliveInterval option, then the work-around for that is to set the shell's TMOUT variable to the desire timeout value. When TMOUT is set, it specifies the number of seconds the shell will wait for a line of input to be entered before closing the shell and thus the SSH session. Note that this means pressing Enter, too, because other typing will not be enough by itself to prevent the timeout and the line must actually be entered for the timer to be reset.

On the server, check for the presence of the SSH_CONNECTION variable, which is usually empty unless currently in an SSH session, to differentiate an SSH session from a local shell. If the account is allowed to make changes to the timeout, then the following can be in the account's own profile, such as ~/.profile,

if [ "$SSH_CONNECTION" != "" ]; then
        # 10 minutes
        TMOUT=600
        export TMOUT
fi

If the account must not be able to change this setting, then it must be in the global profile and made read-only, such as somewhere under /etc/profile.d/,

if [ "$SSH_CONNECTION" != "" ]; then
        # 10 minutes
        TMOUT=600
        readonly TMOUT
        export TMOUT
fi

Both examples above are for Bourne shells, their derivatives, and maybe some other shells. A few shells might have other options available, such as an actual autologout variable found in tcsh(1).

TCP Wrappers, Also Called tcpd(8)

As of 6.7, OpenSSH's sshd(8) no longer supports TCP Wrappers, also referred to as tcpd(8). So this subsection only applies to 6.6 and earlier. Other options that can be used instead of tcpd(8) include packet filters like PF , ipf, NFTables, or even old IPTables. In particular, the Match directive for the OpenSSH server supports filtering by CIDR address. Use these instead and keep in mind the phrase "equivalent security control" which can smooth out hassles caused by security auditors which may still have a "tcpwrappers" checkbox left over from the 1990s on their worksheets.

The tcpd(8) program was an access control utility for incoming requests to Internet services. It was used for services that have a one-to-one mapping to executables, such as sshd(8), and which have been compiled to interact with it. It checked first a whitelist (/etc/hosts.allow) and then a blacklist (/etc/hosts.deny) to approve or deny access. The first pattern to match any given connection attempt was used. The default in /etc/hosts.deny was to block access, if no rules match in /etc/hosts.allow:

sshd: ALL

In addition to access control, tcpd(8) can be set to run scripts using twist or spawn when a rule is triggered. spawn launches another program as a child process of tcpd(8). From /etc/hosts.allow:

sshd: .example.org : allow
sshd: .example.com : spawn \
	/bin/date -u +"%%F %%T UTC from %h" >> /var/log/sshd.log : allow

The variable %h expands to the connecting clients host name or ip number. The manual page for hosts_access(5) includes a full description of the variables available. Because the program in the example, date, uses the same symbol for variables, the escape character (%) must be escaped (%%) so it will be ignored by tcpd(8) and get passed on to date correctly.

twist replaces the service requested with another program. It is sometimes used for honeypots, but can really be used for anything. From /etc/hosts.deny:

sshd: .example.org : deny
sshd: ALL : twist /bin/echo "Sorry, fresh out." : deny

With TCP Wrappers the whitelist /etc/hosts.allow is searched first, then the blacklist /etc/hosts.deny. The first match is applied. If no applicable rules are found, then access is granted by default. It should not be used any more and the better alternatives used instead. See also the Match directive in sshd_config(5) about CIDR addresses or else the AllowUsers and DenyUsers directives.

Using TCP Wrappers To Allow Connections From Only A Specific Subnet

It was possible to use TCP Wrappers just set sshd(8) to listen only to the local address and not accept any external connections. That was one way. To use TCP Wrappers for that, put a line in /etc/hosts.deny blocking everything:

sshd: ALL

And add an exception for the in /etc/hosts.allow by designating the ip range using CIDR notation or by domain name

sshd: 192.0.32.0/20

The same method can be used to limit access to just the localhost (127.0.0.0/8) by adding a line to /etc/hosts.allow:

sshd: 127.0.0.0/8

Again, the best practice is to block from everywhere and then open up exceptions. Keep in mind that if domains are used instead of IP ranges, DNS entries must be in order and DNS itself accessible. However, the above is of historical interest only. The same kind of limitations are better done by setting sshd_config(5) accordingly and instead of using TCP Wrappers utilize the Match directive in the OpenSSH server or the packet filter in the operating system itself.

The Extended Internet Services Daemon (xinetd)

The Extended Internet Services Daemon, xinetd(8), can provide many kinds of access control. That includes but is not limited to name, address or network of remote host, and time of day. It can place limits on the number of services per each service as well as discontinue services if loads exceed a certain limit.

The super-server listens for incoming requests and launches sshd(8) on demand, so it is necessary to first stop sshd(8) from running as standalone daemon. This may mean modifying System V init scripts or Upstart configuration files. Then make an xinetd(8) configuration file for the service SSH. It will probably go in /etc/xinetd.d/ssh The argument -i is important as it tells sshd(8) that it is being run from xinetd(8).

service ssh
{
	socket_type     = stream
	protocol        = tcp
	wait            = no
	user            = root
	server          = /usr/sbin/sshd
	server_args     = -i
	per_source      = UNLIMITED
	log_on_failure  = USERID HOST
	access_times    = 08:00-15:25
	banner          = /etc/banner.inetd.connection.txt
	banner_success  = /etc/banner.inetd.welcome.txt	
	banner_fail     = /etc/banner.inetd.takeahike.txt

	# log_on_success  = PID HOST DURATION TRAFFIC EXIT
	# instances       = 10
	# nice            = 10
	# bind            = 192.168.0.100
	# only_from       = 192.168.0.0
	# no_access       = 192.168.54.0
	# no_access       += 192.168.33.0
}

Finally, reload the configuration by sending SIGHUP to xinetd(8).