DoS: looking at open connections

Here is a command line to run on your server if you think your server is under attack. It prints our a list of open connections to your server and sorts them by ammount.

RedHat: netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n

BSD: netstat -na |awk ‘{print $5}’ |cut -d “.” -f1,2,3,4 |sort |uniq -c |sort -n

You can also check for connections by running the following command.
netstat -plan | grep :80 | awk ‘{print $4 }’ | sort -n | uniq -c | sort

These are few step to be taken when you feel the server is under attack:
——————————————————————————-
Step 1: Check the load using the command “w”.
Step 2: Check which service is utilizing maximum CPU by “nice top”.
Step 3: Check which IP is taking maximum connection by netstat -anpl|grep :80|awk {‘print $5’}|cut -d”:” -f1|sort|uniq -c|sort -n
Step 4: Then block the IP using firewall (APF or iptables “apf -d < IP>” )
——————————————————————————-

You can also implement security features in your server like:

1) Install apache modules like mod_dosevasive and mod_security in your server.
2) Configure APF and IPTABLES to reduce the DDOS
3) Basic server securing steps :
===============================
http://www.linuxdevcenter.com/pub/a/linux/2006/03/23/secure-your-server.html?page=1
===============================
4) Configure sysctl parameters in your server to drop attacks.

You can block the IP which is attacking your server using Ipsec from command prompt.
=========
>> netsh ipsec static add filterlist name=myfilterlist
>> netsh ipsec static add filter filterlist=myfilterlist srcaddr=a.b.c.d dstaddr=Me
>> netsh ipsec static add filteraction name=myaction action=block
>> netsh ipsec static add policy name=mypolicy assign=yes
>> netsh ipsec static add rule name=myrule policy=mypolicy filterlist=myfilterlist filteraction=myaction
========

How do I permit specific users SSH access?

We will be primarily working with one configuration file in this article:

  • OpenSSH/etc/ssh/sshd_config

OpenSSH

For locking down which users may or may not access the server you will want to look into one, or more, of the following directives:

User/Group Based Access

AllowGroups

This keyword can be followed by a list of group name patterns, separated by spaces.If specified, login is allowed only for users whose primary group or supplementary group list matches one of the patterns.`*' and `?' can be used as wildcards in the patterns.Only group names are valid; a numerical group ID is not recognized.By default, login is allowed for all groups.

AllowUsers

This keyword can be followed by a list of user name patterns, separated by spaces.If specified, login is allowed only for user names that match one of the patterns.`*' and `?' can be used as wildcards in the patterns.Only user names are valid; a numerical user ID is not recognized.By default, login is allowed for all users.If the pattern takes the form USER@HOST then USER and HOST are separately checked, restricting logins to particular users from particular hosts.

DenyGroups

This keyword can be followed by a list of group name patterns, separated by spaces.Login is disallowed for users whose primary group or supplementary group list matches one of the patterns.
`*' and `?' can be used as wildcards in the patterns.Only group names are valid; a numerical group ID is not recognized. By default, login is allowed for all groups.

DenyUsers

This keyword can be followed by a list of user name patterns, separated by spaces.Login is disallowed for user names that match one of the patterns.`*' and `?' can be used as wildcards in the patterns.Only user names are valid; a numerical user ID is not recognized.By default, login is allowed for all users.
If the pattern takes the form USER@HOST then USER and HOST are separately checked, restricting logins to particular users from particular hosts.

The first thing to do is backup the original configuration file:

cp /etc/ssh/sshd_config /etc/ssh/sshd_config{,.`date +%s`}

We will now need to edit the configuration file with your favorite editor (vi/vim/ed/joe/nano/pico/emacs.)

An example of only allowing two specific users, admin and bob, to login to the server will be:

/etc/ssh/sshd_config:

AllowUsers admin bob

Ifyou would like to more easily control this for the future then you can create a Group on the server that will be allowed to login to the server, adding individual users as needed (replace username with the actual user):

shell:

groupadd –r sshusers

usermod –a –G sshusers username

With this we will no longer be using AllowUsers but AllowGroups

/etc/ssh/sshd_config:

AllowGroups sshusers

The alternatives to these directives are DenyGroups and DenyUsers which perform the exact opposite of the aforementioned AllowGroups and AllowUsers.
When complete you will want to make sure that sshd will read in the new configuration without breaking.

/usr/sbin/sshd –t

echo $?

We will want to see a 0 following the “echo $?’’ command.Otherwise we should also see an error stating what the erroneous data is:

sshd_config: line 112: Bad configuration option: allowuser
sshd_config: terminating, 1 bad configuration options

After verification we will simply need to restart sshd.This can be performed via many different methods, for which we will assume a sysv-compatible system:

/etc/init.d/sshd restart

Make sure to not disconnect your ssh session but create a new one as a ‘just incase’.
Verify that you can perform any required actions with this user(eg: su into root if you are not allowing root logins.)

Chrootkit help

SSH as admin to your server. DO NOT use telnet, it should be disabled anyways.

#Change to root
su –

#Type the following
wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz

# Check the MD5 SUM of the download for security:
ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.md5

md5sum chkrootkit.tar.gz

#Unpack the tarball using the command
tar xvzf chkrootkit.tar.gz

#Change to the directory it created
cd chkrootkit*

#Compile by typing
make sense

#To use chkrootkit, just type the command
./chkrootkit

#Everything it outputs should be ‘not found’ or ‘not infected’…

Important Note: If you see ‘Checking `bindshell’… INFECTED (PORTS:  465)’ read on.
I’m running PortSentry/klaxon. What’s wrong with the bindshell test?
If you’re running PortSentry/klaxon or another program that binds itself to unused ports probably chkrootkit will give you a false positive on the bindshell test
(ports 114/tcp, 465/tcp, 511/tcp, 1008/tcp, 1524/tcp, 1999/tcp, 3879/tcp, 4369/tcp, 5665/tcp, 10008/tcp, 12321/tcp, 23132/tcp, 27374/tcp, 29364/tcp, 31336/tcp,
31337/tcp, 45454/tcp, 47017/tcp, 47889/tcp, 60001/tcp).

#Now,
cd ..
#Then remove the .gz file
rm chkrootkit.tar.gz

Daily Automated System Scan that emails you a report

While in SSH run the following:
pico /etc/cron.daily/chkrootkit.sh

Insert the following to the new file:
#!/bin/bash
cd /yourinstallpath/chkrootkit-0.42b/
./chkrootkit | mail -s “Daily chkrootkit from Servername” [email protected]

Important:
1. Replace ‘yourinstallpath’ with the actual path to where you unpacked Chkrootkit.
2. Change ‘Servername’ to the server your running so you know where it’s coming from.
3. Change ‘[email protected]’ to your actual email address where the script will mail you.

Now save the file in SSH:
Ctrl+X then type Y

Change the file permissions so we can run it
chmod 755 /etc/cron.daily/chkrootkit.sh

Now if you like you can run a test report manually in SSH to see how it looks.
cd /etc/cron.daily/

./chkrootkit.sh

You’ll now receive a nice email with the report! This will now happen everyday so you don’t have to run it manually.

Rootkit help

RootKit — Spyware and Junkware detection and removal tool

Go to Rootkit Hunter homepage, and download the latest release. http://www.rootkit.nl/projects/rootkit_hunter.html

## Get the latest source and untar
# cd /usr/src/utils
# wget http://downloads.rootkit.nl/rkhunter-<version>.tar.gz
# tar xfz rkhunter-*.gz
# cd rkhunter
# ./installer.sh
## run rkhunter
# rkhunter -c
Setup automatic protection on System Reboot
## Edit /etc/rc.d/rc.local
##      (or similar file depending on Linux version)
## Add the following lines at the bottom of the file

/usr/local/sbin/apf –start
/usr/local/ddos/ddos.sh -c

URL injections information

URL Injection attacks typically mean the server for which the IP address of the attacker is bound is a compromised server.
Please check the server behind the IP address above for suspicious files in /tmp, /var/tmp, /dev/shm, along with checking the process tree (ps -efl or ps -auwx).
You may also want to check out http://www.chkrootkit.org/ and http://www.rootkit.nl/ as tools which should be used in addition to checking the directories and process tree.
Please use “ls -lab” for checking directories as sometimes compromised servers will have hidden files that a regular “ls” will not show.

(see http://en.wikipedia.org/wiki/Remote_File_Inclusion )

1) Installing some apache modules like mod_security and configuring it to prevent $GET requests (this is what happened from your server this time).
2) In order to prevent URL injection you can also :
# Turn off fopen url wrappers
# Disable wget / fetch / lynx binaries
3) Make use of all the utilities provided to you in the Security section of your WHM
4) You can also follow the steps outlined at : http://www.topwebhosts.org/tools/apf-bfd-ddos-rootkit.php
5) Schedule regular security audits on a timely basis – either monthly or weekly – where you can run chkrootkit and rkhunter and scan for vulnerabilities.

Securing the TMP Partition and Tracking Hacks

Are your temp partitions putting out behind your back? Anyone who’s ever administered a Linux server would know the risk of leaving the /tmp directory unsecured, moreso on a webserver that is shared among multiple websites.

The tmp directory is world-writeable and used by a majority of services on a machine — including the storage of PHP and MySQL session files. One issue that I’ve seen on older servers is that one customer’s poorly-coded website would get exploited and end up downloading a file into the /tmp directory, then have that hack file executed on the server as a nobody-owned process. Hack processes are the easiest to find, as they always have a little something extra.

This should go without saying, but hack processes can run from almost anywhere, not just /tmp. I’m mainly bringing up the tmp directory because it’s the most targeted location for hack files if it isn’t secured properly.

Identifying Hack Processes

To start out, log into your server as root as issue the following command to see all the nobody-owned processes running on the machine. This is assuming that Apache on your webserver runs as ‘nobody’:

root@localhost[~] ps -efw |grep nobody |more

A majority of the output will reflect legitimate Apache processes:

nobody 8748 25841 0 21:35 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL
nobody 8785 25841 0 21:35 ? 00:00:01 /usr/local/apache/bin/httpd -DSSL
nobody 8988 25841 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL

You’ll see that they all have the same parent process ID of ‘25841′ and all of the commands look about the same. However, you may see something that looks like this:

nobody 15707 26407 0 11:18 ? 00:00:00 [sh <defunct>]
nobody 15717 1 0 11:18 ? 00:00:04 /usr/bin/perl
nobody 13016 1 0 14:14 ? 00:00:00 /usr/bin/perl

nobody 8988 1 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL -d3

All of these are hack processes. A few things to look for:

– Perl and shell (sh) processes on most systems run as the user that is executing them, not ‘nobody’. If this is true on your system, a nobody-owned perl process is probably a hack

– The parent ID of the process is different than all the others of the same service — these can be easily faked

– For Apache, an extra little switch on the end of the command (../httpd -DSSL -d3) – ‘d3′ is usually the name of the hack file that is being executed through Apache (not always).

Now I know you probably have the urge to kill -9 it, but don’t. Killing a hack process before determining what it is or where it came from is only going to let it happen again.

Tracking the process:

Once you know which processes should not be running, you need to know where they came from. The *easiest* way to track a process is through the lsof command. If you take the process ID (not the parent) and run an lsof it will tell you everything you want to know:

lsof -p 8748

From here I can see that this process is running from a folder in a user’s account. All I need to do now is kill it off, then remove the hack files from that user’s directory and let them know to update their shit.

TMP Directory Hacks

I’ve seen my fair share, but many hack processes will be spawned from the temp partitions of the server (like /tmp, /tmpDIR, /dev/shm), as these are usually set to 777 to allow programs to store their temporary files. When you first look in your TMP directory you’re going to see a lot of stuff — this is normal, but you’ll want to look for nobody-owned files that stand out. Just a hint, some hacks will try to trick you with their names. I have seen many that are named “…” or “httpd” to make them look legitimate, but I know that httpd (the service that runs Apache) should not be in the tmp directory…and that’s what makes it stand out.

Securing the tmp Partition

I’m not going to go too far into this here, because there’s already a very good website with this information available.

If you are on a cPanel server, you should run /scripts/securetmp in addition to following the instructions above.

Installing IonCube loader with Zend Optimizer – cPanel

This is a common request we get for Ioncube to be installed. It’s generally not an issue, but when you factor in other optimization plugins like Zend and eAccelerator, a common misconception is that the three don’t get along. It’s very easy to install Ioncube into a PHP installation that already has Zend and eAccelerator.

This tutorial is specific to cPanel, assuming that you are using php 5.2.x with Zend 3.x.x.

If you need help installing eAccelerator, you can see this tutorial. For help with installing Zend, you can go here. The versions in both these tutorials are outdated, so you’ll probably want to apply the instructions to the newest versions available.
Go to http://www.ioncube.com/loader_download.php and pick your download. This example assumes that you are using php 5.2.

cd /usr/src
wget http://downloads2.ioncube.com/loader_downloads/ioncube_loaders_lin_x86.tar.gz
tar -xvzf ioncube_loaders_lin_x86.tar.gz
cd ioncube

Copy the loader config to the user’s public_html or another location where you can access it from a browser.

cp ioncube-loader-helper.php /home/username/public_html

Now in your browser go to the loader file that you just copied. This file will tell you exactly which extension you need to use. Choose the ‘php.ini Installation Instructions’ link, and you should see something like this after the php config output:

zend_extension = /<path>/ioncube_loader_lin_5.2.so

Move the ioncube directory to a more permanent location:

mv /usr/src/ioncube /usr/local
chown -Rf root:root /usr/local/ioncube

Edit the php.ini and add look for this section (may not be exact):

[Zend]
zend_extension_manager.optimizer=/usr/local/Zend/lib/Optimizer-3.0.1
zend_extension_manager.optimizer_ts=/usr/local/Zend/lib/Optimizer_TS-3.0.1
zend_optimizer.version=3.0.1
zend_extension=/usr/local/Zend/lib/ZendExtensionManager.so
zend_extension_ts=/usr/local/Zend/lib/ZendExtensionManager_TS.so

Above this section, add this line:

zend_extension=/usr/local/ioncube/ioncube_loader_lin_5.2.so

Of course, make sure that the .so file is the one that the loader helper told you to use! After that is added, STOP and then START Apache to make sure that it’s loading. You should now see IonCube in your phpinfo file.

If you’re using eAccelerator, you shouldn’t need to change the location of the plugin loader in your php.ini.

Note that if Apache doesn’t start, it’s probably because of the order in which you have Zend and ioncube loading.  The lines for Ioncube should be above those for Zend optimizer.

Lastly, you should test your IonCube installation to make sure that it can decode its own files. In the original ‘ioncube’ directory that you moved, there’s a test ‘ioncube-encoded-file.php’ file that you can load through a browser to make sure that it works.

eAccelerator on PHP 5

Here is what I did to get eAccelerator 0.9.5 running on PHP 5 on my V-Dedicated server. I did the exact steps when I had PHP 4 running the only difference was that with PHP 4 I used the 0.9.4 install of eAccelerator instead of 0.9.5. Everything was done via SSH access.

1. cd /usr/local/src/
2. wget http://bart.eaccelerator.net/source/0.9.5/eaccelerator-0.9.5.tar.bz2
3. bunzip eaccelerator-0.9.5.tar.bz2
3. tar -xvf eaccelerator-0.9.5.tar
4. cd eaccelerator-0.9.5
5. Run the following command inside that directory /usr/local/bin/phpize. You should get output like the following

Configuring for:
PHP Api Version: 20041225
Zend Module Api No: 20050922
Zend Extension Api No: 220051025

6. Run the next commands to get it all configured to work with your system and compiled/installed

./configure –enable-eaccelerator=shared –with-php-config=/usr/local/bin/php-config
make
make install

7. After running the make install the system will tell you know where it installed the module to. Take note of that directory, it should be something much like the following

Installing shared extensions: /usr/local/lib/php/extensions/no-debug-non-zts -20050922/

you will need this information for the php.ini else the module will not load.

8. Now you need to create the eaccelerator cache directory and give it the correct permissions.

mkdir /tmp/eaccelerator
chmod 0777 /tmp/eaccelerator

9. This is reallt the last main step, Next you need to add the following into your php.ini so that the module loads.

;;;;;;;;;;;;;;;;
; eAccelerator ;
;;;;;;;;;;;;;;;;
extension=”/usr/local/lib/php/extensions/no-debug-non-zts-20050922/eaccelerator.so”
eaccelerator.shm_size=”16″
eaccelerator.cache_dir=”/tmp/eaccelerator”
eaccelerator.enable=”1″
eaccelerator.optimizer=”1″
eaccelerator.check_mtime=”1″
eaccelerator.debug=”0″
eaccelerator.filter=””
eaccelerator.shm_max=”0″
eaccelerator.shm_ttl=”0″
eaccelerator.shm_prune_period=”0″
eaccelerator.shm_only=”0″
eaccelerator.compress=”1″
eaccelerator.compress_level=”9″

Of course make sure to change out extension=”/usr/local/lib/php/extensions/no-debug-non-zts-20050922/eaccelerator.so” with the exact location of your module as told to you after the make install command.

10. Restart Apapche and it should be all set. You can load up your phpinfo page and you should see that the eAccelerator is loaded on the system

Preventing DDOS Attacks with Mod_Evasive

Denial of Service attacks are among the oldest yet most common form of attacking a server. Most system administrators have had to deal with DOS attacks taking down a server, router, or other networking device and know how difficult they can be to prevent.

Mod_evasive is an Apache module that limits the number of Apache connections to the server at once, and blocks an offending IP for a specified amount of time. This tutorial will show you how to install mod_evasive on your system.

1. Install:

wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
tar -xvzf mod_evasive_1.10.1.tar.gz
cd mod_evasive_1.10.1
/usr/local/apache/bin/apxs -cia mod_evasive.c

2. Configure:

Once the module is compiled, add these lines to httpd.conf and stop Apache completely before starting it up again:

<IfModule mod_evasive.c>
DOSHashTableSize 3097
DOSPageCount 6
DOSSiteCount 50
DOSPageInterval 2
DOSSiteInterval 2
DOSBlockingPeriod 600
</IfModule>

Below is an explanation of these settings:
DOSHashTableSize – Size of the hash table. The greater this setting, the more memory is required – faster

DOSPageCount – Max number of requests for the same page within the ‘DOSPageInterval’ interval

DOSSiteCount – Max number of requests for a given site, uses the ‘DOSSiteInterval’ interval.

DOSPageInterval – Interval for the ‘DOSPageCount’ threshold in second intervals.

DOSSiteInterval– Interval for the ‘DOSSiteCount’ threshold in second intervals.

DOSBlockingPeriod – Blocking period in seconds if any of the thresholds are met. The user will recieve a 403 (Forbidden) when blocked, and the timer will be reset each time the site gets hit when the user is still blocked.