Reverse Engineering/Basic Security
Single user systems
There are several simple ways to defend a system against malicious attack, although the measures needed vary with the use of the system, with a single-user server being the easiest to secure. The most obvious method, which is applicable to almost all types of systems, is to only run the servers/daemons which you need. If you run a thousand and one different daemons just because they came with your distro, you have to keep them up to date. For example, if you are only running 2 or 3, you can consistently check for updates, confident that you didn't forget about that tftp server you never actually use. It should go without saying that you should keep these things up to date, since the sooner you can patch/upgrade a vulnerability in a daemon or the kernel, the less vulnerable you will be to attack. A large part of it is "security through obscurity"; people or programs have to find out about it before it can be attacked.
These are the only things needed to reasonably defend a single-user system; however, when you have many user accounts, even if you trust the people given them ( you can never \really\ trust a user like this but lets not be paranoid), there are extra security measures which need to be implemented. These include strict privileges for users accounts and a good password policy. It is one thing for an attacker to find a vulnerability in a program, but if they are simply allowed to use a dangerous program with a compromised account you will be under much more risk. If you have a complicated system where files need sharing between users, remember that you can create a separate group for these users and use the group permissions to allow the users access without allowing world period between changing passwords, a user will use very simple passwords or worse use a Post-it on their monitor.
Attackers often use something called root-kits which make it seem like there is nothing untoward running despite the fact that they have control of your system. They do this by replacing common system utilities with versions which don't report the whole truth; for example, a version of ps which doesn't display processes with "0wned" in their name. These are called root kits as you have to be root (or a similar all powerful user) to install these on a system, wikipedia covers root kits. You can defend against a root kit using a Intrusion Detection System (IDS); however if you suspect that something unpleasant is happening, you can check using a known clean version of the system utilities. These can be found conveniently bundled in a package called busybox which can be found at http://www.busybox.net/download.html.
In such cases it is also possible to create ahead of time statically linked versions of all of the utilites and to then write them to a CDROM that can be used in such situations.
However these days most common root-kits are kernel-based. Which means that root-kits infect (like a parasite) the core of the operating system where all basic operations are performed and where hardware is accessed. The operating system kernel manages process scheduling, memory management, device access and device drivers. When the kernel is infected by a root-kit (backdoor), for example like a device driver, through a loadable kernel module, there is no way to tell if a system is compromised because all input/output in the system can be faked. When the rootkit is in place memory usage, cpu usage, running processes and files and directories can all be hidden.
There is no remedy in this case in a running system. The only way to check a system's integrity in such cases is by booting the system from another read-only medium which is known to be trusted and to verify the integrity of the installed system through this means. It is possible to install integrity checking software such as tripwire or aide (advanced intrusion detection system) which builds a database of signatures of installed files. At a later point in time this database can be used to check if files have been changed and it is then possible to tell if a change was authorized or not.
Attackers could leave more than one means to get back in through the use of backdoors into the system so always make sure you removed all of them. If you have no database of signatures you are usually unable to be certain that you have found all backdoors and then it is recommended to reinstall the system to regain a known clean system.