Norton blacklisted my website and it is CLEAN!

Norton for a long time used to show clients site as blacklisted, I registered for https://safeweb.norton.com/ and sent a review request........ days passed by with no reply.
Scanned site at sitecheck.sucuri.net which too showed clean but blacklisted at Norton.  
Later found a quick solution on Norton forums :

Submit category change request with WebPulse:

https://sitereview.symantec.com/#/

 Please enter a valid URL for the review process -
> valid URL for review -
https://ecoinsee.org/
> Filtering Service - select 'Norton Safe Web'-
> Your suggested category - 'Select a Category'-
> Comments - add 'Safe Web Orange Caution with 0 Threats'-


After few hours I see the site now clean.


https://safeweb.norton.com/report/show?url=ecoinsee.org

CentOS 7 :: /var/log/messages , cron , secure log files empty

Today I found a cPanel CentOS 7 server with empty log files :

-rw------- 1 root root 0 Aug 18 05:57 cron
-rw------- 1 root root 0 Aug 18 05:57 maillog
-rw------- 1 root root 0 Aug 18 05:57 messages
-rw------- 1 root root 0 Aug 18 05:57 secure

Found a quick solution :

rm -f /var/lib/rsyslog/imjournal.state

service rsyslog restart

Server reboot and service restarts did not help however the above worked..

Voila the logs started working…….

Plesk Onyx 17.8 :: Backup Error :: Errcode: 24 – Too many open files

A recently setup Plesk on CentOS 7 Server was showing errors under Backup manager :

Unable to execute SQL: Out of resources when opening file '/tmp/#sql_5226_0.MYI' (Errcode: 24 - Too many open files). SQL query: SHOW FULL COLUMNS IN `catalogindex_minimal_price`

The MySQL configured was default and showed limits as below :

mysql> show global variables like 'open%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| open_files_limit | 1000  |
+------------------+-------+
1 row in set (0.00 sec)

mysql>

To raise the limits I had to create a directory and create a configuration file as below :

cd  /usr/lib/systemd/system
mkdir mysql.service.d
cd mysql.service.d/
nano -w override.conf


# cat override.conf
[Service]
LimitNOFILE=5000
#
systemctl daemon-reload
service mysqld restart

Once this was applied the Backups worked fine. You may raise the limits as required.


Parallels Guest Tool Install Error :: FATAL: modpost: GPL-incompatible module prl_eth.ko uses GPL-only symbol ‘sev_enable_key’

Today faced a issue with Virtuozzo 6 Guest tool install. Got below error while installing guest tools for CentOS.

Start installation or upgrade of Guest Tools
new version of virtuozzo tools
Installed Guest Tools were not found
Perform installation into the /usr/lib/parallels-tools directory
cat: /usr/lib/parallels-tools/kmods/../version: No such file or directory
Start installation of prl_eth kernel module
make: Entering directory `/usr/lib/parallels-tools/kmods'
cd prl_eth/pvmnet && make
make[1]: Entering directory `/usr/lib/parallels-tools/kmods/prl_eth/pvmnet'
make -C /lib/modules/3.10.0-957.1.3.el7.x86_64/build M=/usr/lib/parallels-tools/kmods/prl_eth/pvmnet
make[2]: Entering directory `/usr/src/kernels/3.10.0-957.1.3.el7.x86_64'
LD /usr/lib/parallels-tools/kmods/prl_eth/pvmnet/built-in.o
CC [M] /usr/lib/parallels-tools/kmods/prl_eth/pvmnet/pvmnet.o
LD [M] /usr/lib/parallels-tools/kmods/prl_eth/pvmnet/prl_eth.o
Building modules, stage 2.
MODPOST 1 modules
FATAL: modpost: GPL-incompatible module prl_eth.ko uses GPL-only symbol 'sev_enable_key'
make[3]: *** [__modpost] Error 1
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/usr/src/kernels/3.10.0-957.1.3.el7.x86_64'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/usr/lib/parallels-tools/kmods/prl_eth/pvmnet'
make: *** [installme] Error 2
make: Leaving directory `/usr/lib/parallels-tools/kmods'
Error: could not build kernel modules
Error: failed to install kernel modules
Error during report about failed installation of virtuozzo tools.

Found a fix that works at https://gist.github.com/rudolfratusinski/a4d9e3caff11a4d9d81d2e84abc9afbf

I used below steps :

Rsync used to make sure permissions are intact :

rsync -avz /mnt/cdrom /root/

Extract files to edit :
cd /root/cdrom/kmods/
tar -xzf prl_mod.tar.gz

Edit first File

nano -w /root/cdrom/kmods/prl_eth/pvmnet/pvmnet.c

Change below :

MODULE_LICENSE("Parallels");

to

MODULE_LICENSE("GPL");

Edit Second File

nano -w /root/cdrom/kmods/prl_tg/Toolgate/Guest/Linux/prl_tg/prltg.c

Change below :

MODULE_LICENSE("Parallels");

to

MODULE_LICENSE("GPL");

Once the 2 files are saved zip back the files.

cd /root/cdrom/kmods/
tar -zcvf prl_mod.tar.gz . dkms.conf Makefile.kmods

Now try the install which should complete without error

cd /root/cdrom/
./install

OpenVZ 7 Upgrade Error !

While upgrading OpenVZ server I got an error as below :

====
--> Finished Dependency Resolution
Error: Package: 10:qemu-kvm-vz-2.9.0-16.3.vz7.8.x86_64 (factory)
Requires: libspice-server.so.1(SPICE_SERVER_0.12.5)(64bit)
You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
grub2-theme-openvz-1.4-1.vl7.noarch has installed conflicts grub2-theme-openvz: grub2-theme-openvz-1.4-1.vl7.noarch

====

This is a known bug and has been patched by OpenVZ Team and below is the fix :

Update vzlinux-release
yum update vzlinux-release
Enable virtuozzolinux-factory repo – edit /etc/yum.repos.d/vzlinux.repo or launch
yum-config-manager --enable virtuozzolinux-factory
Actually launch “yum update”
yum update

This fixes the bug. 🙂

Reference : https://bugs.openvz.org/browse/OVZ-6924

Disable MySQL strict mode!

We recently migrated WHMCS to latest CentOS 7 server with cPanel. The WHMCS ClientArea showed White screen when “Client logins were used”. We found that the server had MySQL strict mode enabled which was causing this problem.

I disabled MySQL strict mode as below :

Edited /usr/my.cnf and changed below value :

sql_mode=NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER

Restarted MySQL :

/scripts/restartsrv mysql

voila this worked…… earlier I was trying to change sql_mode under /etc/my.cnf however found MySQL is using different location /usr/my.cnf

WordPress :: xmlrpc.php Attack!

WordPress is the most targeted CMS nowadays and needs to be updated regularly. Recently I have seen attacks on wordpress xmlrpc.php using POST requests and the attack is large enough to take down / freeze the server.

The top or ps aufxw shows most of the xmlrpc.php requests as below :

==========
linuxbabu 4414 1.3 0.4 85512 35544 ? R 20:57 0:00 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3876 0.0 0.1 25936 10852 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
linuxbabu 4413 1.3 0.4 85512 35672 ? R 20:57 0:00 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3877 0.0 0.1 25936 10852 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
linuxbabu 4418 1.3 0.3 82936 33472 ? R 20:57 0:00 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3878 0.0 0.1 25936 10764 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
linuxbabu 4381 1.3 0.5 91580 42368 ? R 20:56 0:01 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3879 0.0 0.1 25936 10768 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL

89.248.168.164 - - [30/Jul/2014:16:51:27 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:03:11 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:03:09 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:06:38 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:09:30 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:16:27 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:16:26 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:35:03 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:37:30 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)
==========

First thing you would think is why not delete xmlrpc.php, NO that will generate 404 error and all attack will process the 404 page that too wont help reduce the load.

Here is a quick way to get control of the server :

Add below to .htaccess :

#redirect xmlrpc.php attack to attacking servers
Redirect 301 /xmlrpc.php http://127.0.0.1

You would see a drastic load change and load back to normal and get back your smile 🙂

cPanel :: upcp failed, exited with code 25

I noticed below error while updating cPanel to latest one of the server:

Running `/usr/local/cpanel/scripts/updatenow --upcp --log=/var/cpanel/updatelogs/update.9320814564.log` failed, exited with code 25 (signal = 0)

Below was the OS installed on the server :

root@linuxbabu [~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
root@linuxbabu [~]#

Further found that the cPanel config had the OS information missing as below :

root@linuxbabu [~]# cat /var/cpanel/sysinfo.config
# This values in this file are calculated and updated if necessary nightly. If you wish to override
# these values, populate lock= with a comma delimited list of keys you don't want updated.
# This will allow you to change those values to suit your needs.
#
# Example: if you set lock like this, then rpm_dist and rpm_arch will not be updated nightly:
# lock=rpm_dist,rpm_arch
#
ises=2
lock=
release=6.5
rpm_arch=x86_64
rpm_dist=centos
rpm_dist_ver=unknown
root@linuxbabu [~]#

I corrected the OS rpm_dist_ver under the config file and the upcp worked fine :

root@linuxbabu [~]# cat /var/cpanel/sysinfo.config | grep rpm_dist_ver
rpm_dist_ver=6
root@linuxbabu [~]#

cPanel : SSHD Rootkit

We have been seeing this SSHD rootkit from time to time with much improved versions 🙂

One could see number of SSH processes on the server with nothing under process details. There is a huge discussion thread at WHT www.webhostingtalk.com/showthread.php?t=1235797

The current one which I have seen today was with keyutils-libs legit version name. However the “Signature” was missing.

root@linuxbabu [/var/log]# ls -la /lib64 | grep libkeyutils
lrwxrwxrwx 1 root root 18 Jun 22 2012 libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 35320 Jun 22 2012 libkeyutils.so.1.3*

root@linuxbabu [/var/log]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 14 Jan 2014 04:56:45 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 59336 License: GPLv2+ and LGPLv2+
Signature : (none)
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.

I matched the size from a clean server and found different in the file size of libkeyutils.so.1.3.

So I reinstalled keyutils-libs replacing the file using yum.

yum reinstall keyutils-libs -y

After reinstall I could see the file size changes and also the Signature shown was correct :

root@linuxbabu [~]# ls -la /lib64/libkeyutils*
lrwxrwxrwx 1 root root 18 Feb 25 01:03 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 10192 Jun 22 2012 /lib64/libkeyutils.so.1.3*
root@linuxbabu [~]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 25 Feb 2014 01:03:47 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 36624 License: GPLv2+ and LGPLv2+
Signature : RSA/SHA1, Sun 24 Jun 2012 05:18:51 PM CDT, Key ID 0946fca2c105b9de
Packager : CentOS BuildSystem
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.
root@linuxbabu [~]#

Hope this helps to disable the rootkit and avoid further damage to the server.

Another MySQL daemon already running with the same unix socket.

After upgrading MySQL binaries mysqld will not start at all any more and shows below error :

CT-1977-bash-4.1# service mysqld start
Another MySQL daemon already running with the same unix socket.
Starting mysqld: [FAILED]
CT-1977-bash-4.1#

MySQL service does not shut down gracefully during the OS reboot, leaving the old /var/lib/mysql/mysql.sock such that mysqld will not start up. Some people were able to reproduce this error on a CentOS 6.5 KVM guest virtual system by rebooting the host CentOS 6.5 system. CentOS is supposed to gracefully shut down the guest systems, but this seems to be failing in the case of mysqld.

Confirmed Red Hat Linux 6.5 bug – https://bugzilla.redhat.com/show_bug.cgi?id=1037650

Issue discussion on MySQL bug tracker – http://bugs.mysql.com/bug.php?id=71086

Simple steps to reproduce this issue:

service mysqld start
killall -9 mysqld_safe mysqld
service mysqld start

A quick way to restart MySQL is as below :
Remove socket file and restart mysql:
CT-1977-bash-4.1# ls -la /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Jan 8 20:13 /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# rm /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

A work around for the issue is to modify ‘/etc/init.d/mysqld’ script:

Make a backup copy of the startup script.
cp -p /etc/init.d/mysqld /etc/init.d/mysqld.orig

Edit the file /etc/init.d/mysqld to look as below :
=========================
# if fuser "$socketfile" &>/dev/null ; then
# echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
# action $"Starting $prog: " /bin/false
# return 1

# We check if there is already a process using the socket file,
# since otherwise this init script could report false positive
# result and mysqld_safe would remove the socket file, which
# actually uses a different daemon.
if fuser "$socketfile" &>/dev/null ; then
echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
action $"Starting $prog: " /bin/false
return 1
fi
=========================

Now restart MySQL :
CT-1977-bash-4.1# service mysqld start
Starting mysqld: [ OK ]
CT-1977-bash-4.1#