A buffer overflow is the most common and the most serious threat to Linux/Unix operating systems. Buffer overflows occur when code running in unprotected memory in a buffer overwrites memory in an adjacent location. For example, a string of information, say 20 bytes, is sent to a 16-byte buffer, which can’t accommodate that string. Linux/Unix is written in C and there are memory-safety issues to consider, it could occur as a result of programming errors if memory boundary checking is inefficient or if boundary checking in a process isn’t present.
Buffer overflow becomes a security issue when an attacker manages to maliciously insert code into the memory of a running process, the attacker may then be able to acquire the privileges of that process. Generally buffer overflow is stack based where malicious code placed on a stack changes the return pointer and consequentially program flow, and heap based where overflowing buffers on lower parts of the heap result in unpredictable effects on the rest of the heap.
When trying to prevent buffer overflow, there are few preventative measures you can take. Of course, careful programming should ensure that attackers aren’t made aware of where privilege processes are running from. So analysis of source code for potential vulnerabilities is highly recommended. However, given that programmers are human, buffer overflows are always going to be a problem.
It’s important to disable stack execution and randomize stack location because this will make it more difficult for attackers to locate overflows and inject malicious code. When possible, if you are writing programs, use languages that have memory protection features, for example, Java or C#.
In C and C++, there are no automatic bounds checking on the buffer, which means a user can write past a buffer. For example:
int main () {
int buffer[10];
buffer[20] = 10;
}
The above C program is a valid program, and every compiler can compile it without any errors. However, the program attempts to write beyond the allocated memory for the buffer, which might result in unexpected behavior.
It’s always a good idea to perform runtime checking to ensure that buffer overflow isn’t occurring. In certain operating systems, you can enforce stack protection. With stack protection, you can modify your kernel to ensure that code only executes from areas of memory that you have cordoned off and locked. If a malicious user attempts to insert code into the stack-protected memory of a running process, the process will abort. Service will be denied, but the system won’t be compromised. The methods vary depending on the distribution of Linux/Unix or Linux that you are using.
