Are your temp partitions putting out behind your back? Anyone who’s ever administered a Linux server would know the risk of leaving the /tmp directory unsecured, moreso on a webserver that is shared among multiple websites.
The tmp directory is world-writeable and used by a majority of services on a machine — including the storage of PHP and MySQL session files. One issue that I’ve seen on older servers is that one customer’s poorly-coded website would get exploited and end up downloading a file into the /tmp directory, then have that hack file executed on the server as a nobody-owned process. Hack processes are the easiest to find, as they always have a little something extra.
This should go without saying, but hack processes can run from almost anywhere, not just /tmp. I’m mainly bringing up the tmp directory because it’s the most targeted location for hack files if it isn’t secured properly.
Identifying Hack Processes
To start out, log into your server as root as issue the following command to see all the nobody-owned processes running on the machine. This is assuming that Apache on your webserver runs as ‘nobody’:
root@localhost[~] ps -efw |grep nobody |more
A majority of the output will reflect legitimate Apache processes:
nobody 8748 25841 0 21:35 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL
nobody 8785 25841 0 21:35 ? 00:00:01 /usr/local/apache/bin/httpd -DSSL
nobody 8988 25841 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL
You’ll see that they all have the same parent process ID of ‘25841′ and all of the commands look about the same. However, you may see something that looks like this:
nobody 15707 26407 0 11:18 ? 00:00:00 [sh <defunct>]
nobody 15717 1 0 11:18 ? 00:00:04 /usr/bin/perl
nobody 13016 1 0 14:14 ? 00:00:00 /usr/bin/perl
nobody 8988 1 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL -d3
All of these are hack processes. A few things to look for:
– Perl and shell (sh) processes on most systems run as the user that is executing them, not ‘nobody’. If this is true on your system, a nobody-owned perl process is probably a hack
– The parent ID of the process is different than all the others of the same service — these can be easily faked
– For Apache, an extra little switch on the end of the command (../httpd -DSSL -d3) – ‘d3′ is usually the name of the hack file that is being executed through Apache (not always).
Now I know you probably have the urge to kill -9 it, but don’t. Killing a hack process before determining what it is or where it came from is only going to let it happen again.
Tracking the process:
Once you know which processes should not be running, you need to know where they came from. The *easiest* way to track a process is through the lsof command. If you take the process ID (not the parent) and run an lsof it will tell you everything you want to know:
lsof -p 8748
From here I can see that this process is running from a folder in a user’s account. All I need to do now is kill it off, then remove the hack files from that user’s directory and let them know to update their shit.
TMP Directory Hacks
I’ve seen my fair share, but many hack processes will be spawned from the temp partitions of the server (like /tmp, /tmpDIR, /dev/shm), as these are usually set to 777 to allow programs to store their temporary files. When you first look in your TMP directory you’re going to see a lot of stuff — this is normal, but you’ll want to look for nobody-owned files that stand out. Just a hint, some hacks will try to trick you with their names. I have seen many that are named “…” or “httpd” to make them look legitimate, but I know that httpd (the service that runs Apache) should not be in the tmp directory…and that’s what makes it stand out.
Securing the tmp Partition
I’m not going to go too far into this here, because there’s already a very good website with this information available.
If you are on a cPanel server, you should run /scripts/securetmp in addition to following the instructions above.