Reverse Engineering/Common Solutions

Protection Mechanisms

Not many good protective measures are available to programmers to prevent most overflow vulnerabilities. However, something can be done.

Bounds Checking

New languages such as Java and C# make such a big deal over their "automatic bounds checking" and "memory management" features mainly because they help prevent stack overflows (with a small performance penalty). C programmers however are left to their own devices, and need to explicitly test the bounds on every array. It's tedious, but at least crackers won't break your program, and then you won't get fired from your job.

Canary / Cookie

Some compilers help out by building in a flag value called a canary or cookie on the stack usually just above the pushed frame pointer and the return address (think of the caged bird used in coal mines to detect the buildup of poisonous gases before the workers could get intoxicated).

push CANARY
push ebp
mov ebp, esp
sub esp, 100

Now, when the function wants to return, we can perform the following operation:

add esp, 100
mov esp, ebp
pop ebp
pop ebx ;canary value
cmp ebx, CANARY
jne _STACK_ERROR_FOUND
ret

This way we can detect if the stack has been overwritten, because the Canary value has changed. A predictable Canary value however is vulnerable: attackers that insert that value onto the stack as part of their overflow data elude detection. For this reason most Canary values are randomly generated at run-time. Many Canary values also contain two null characters at the start or end: string copy functions (like strcpy or wcscpy) stop copying data after reaching and writing a null char; if the nulls are instead omitted by the attacker the overflow will be caught.

This method of protection can catch basic overflows, and prevent a function from returning to a modified address and execute arbitrary code. However the subroutine still gets executed – with compromised internal state and variables – since the overflow get detected only when it returns. This can still be exploited by an attacker: for example, a memory pointer variable can be modified to point to an arbitary location. If the subroutine then uses this pointer to write to memory, it could overwrite anything in the program’s address space.

Pointer Sanity

Many heap overflows become effective by overwriting the housekeeping data at the start of the next heap chunk, which normally contains at least one linked list. Allocating or freeing an overwritten chunk can cause data to be written at an arbitary address in memory. Most heap systems now check the data pointed to by linked lists, to ensure that they point at another heap chunk or valid data.

This method of protection is also present in the Microsoft Windows "Structured Exception Handling" routines. Before calling an exception handler (the pointer to which resides on the stack, and can be overwritten), it is first checked to ensure that the routine resides within an executable section of memory. If the handler routine does not, then it is not called.

Safe String Libraries

Because the standard library string functions are the common cause of stack overflows, a number of libraries with "safe" string functions have appeared to try to address this problem. Most of them require an explicit “string length” parameter in their functions’ arguments, and limit the data copied to that amount.

The programmer must obviously still be careful and enter accurate string length values; sloppy programming can still cause trouble.

Exercises

We will leave as an exercise for the reader to write a set of safe string functions, that take a length parameter, and perform simple bounds-checking to prevent overflow. Another option would be to take as an argument a pointer to a "maximum" stack position, and compare pointers to prevent overflow.