Understanding Rootkits

I've been reading a lot about undetectable malware androotkits — and the like —recently. Without a doubt, these attack tools have been iteratively improving over the years. Like most such security "nasties," however, a bit of safe computing goes a long way. Let's explore a bit. First off, let me explain what a rootkit is so we can consider the facts and not get caught up in the hype. A rootkit is a tool, or (more commonly) a collection of tools, that an attacker can install on a compromised computer. The functionality of rootkits vary tremendously from one version to the next, and they're available for just about any computer operating system and architecture in existence today. What they have in common, however, is generally a small set of features:
they hide their (and the attacker's) presence on the compromised computer and
they enable the attacker to log back onto the compromised computer. Most rootkits include tools for removing historical indications the attacker has been on the computer as well, but I think of that as just one aspect of hiding their presence.
A subtle, but important, issue here is that rootkits typically don't have their own collection of tools for providing the attacker with elevated privileges, but they do require those elevated privileges to already be in place in order to install on a victim's computer. So the attacker has to somehow get elevated (e.g., root or administrator) privileges before a rootkit can be installed.

Ever since I first saw a rootkit installed a computer during a system compromise back in the 1994-1995 time frame, I've been watching them and following new rootkit technologies as they've been unleashed. The earliest rootkits accomplished their goals by replacing normal system tools on the victim.s computer with altered versions. Since most of the early rootkits were UNIX-based, their (altered) tools included login, ls, ps, df, netstat, and so on — tools a UNIX user or administrator would routinely run to look at files on a system, processes running in memory, disk utilization and so on. The rootkit versions of these tools did all of these things and more. Specifically, they carried out the features I described above. Pretty soon after these started appearing, the IT Security community got wise and started running login, ls, ps, df, etc., from CD so they could be sure they.re using tools that haven.t been tampered with.
So, the attackers responded by modifying the underlying system shared libraries and leaving the tools intact. So, when ps runs, it returns a list of all the processes on the system, except for those owned by the attacker.not because it has been tampered with, but because the system calls it made returned erroneous information. More recently, rootkits have been installed as kernel loadable modules and such. This has made things increasingly difficult for the IT Security folks to detect rootkits, since they keep going lower (in a software abstraction sense). In fact, with modern microprocessor technology, any software — legitimate or malicious — can pass along deceptive or erroneous data to software that calls it, so long as it is the first in line. In other words, if your software loads first and intercepts system calls, then you can control what others see. If someone else comes along and can find a way to butt into the line, then they can control what others (and you) see. That's the nature of the beast, I'm afraid. So what can we do about it? On the surface, the answer is simple (don't run a rootkit or allow one to be run on your computer), but in practice it's not quite so trivial. However, here are a few things that can help in preventing bad stuff from happening,:
Make judicious use of privileges. Remember the principle of least privilege? Well, you need to put it into practice. If your users run with privileges on their desktops, then the environment is ripe — in fact, it is ideal — for malicious insertion of a rootkit. Users should be able to run software, but not install software. Likewise, when you're logged in as an administrator to do administrative things, that's all you should be doing.
As much as I hate security patches, it's still important to stay up to date with them. Sure, we've all heard this a gazillion times, but unpatched systems provided rootkits with easy avenues of entry to your system. Remember I said most rootkits need to already have privileges in order to install? I'm constantly amazed by how many people don't run some form of Windows Update on their Windows desktops.
Antivirus programs, firewalls, and — those things — are also important layers of security, of course.
Of course, those are just a few things that can be done. The list can't guarantee safety from rootkits and other malware, but it sure can go a long way to reducing the risk, if the recommendations are well-thought-out and implemented.

From Webopedia.com