Disable MySQL strict mode!

We recently migrated WHMCS to latest CentOS 7 server with cPanel. The WHMCS ClientArea showed White screen when “Client logins were used”. We found that the server had MySQL strict mode enabled which was causing this problem.

I disabled MySQL strict mode as below :

Edited /usr/my.cnf and changed below value :

sql_mode=NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER

Restarted MySQL :

/scripts/restartsrv mysql

voila this worked…… earlier I was trying to change sql_mode under /etc/my.cnf however found MySQL is using different location /usr/my.cnf

cPanel :: upcp failed, exited with code 25

I noticed below error while updating cPanel to latest one of the server:

Running `/usr/local/cpanel/scripts/updatenow --upcp --log=/var/cpanel/updatelogs/update.9320814564.log` failed, exited with code 25 (signal = 0)

Below was the OS installed on the server :

root@linuxbabu [~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
root@linuxbabu [~]#

Further found that the cPanel config had the OS information missing as below :

root@linuxbabu [~]# cat /var/cpanel/sysinfo.config
# This values in this file are calculated and updated if necessary nightly. If you wish to override
# these values, populate lock= with a comma delimited list of keys you don't want updated.
# This will allow you to change those values to suit your needs.
#
# Example: if you set lock like this, then rpm_dist and rpm_arch will not be updated nightly:
# lock=rpm_dist,rpm_arch
#
ises=2
lock=
release=6.5
rpm_arch=x86_64
rpm_dist=centos
rpm_dist_ver=unknown
root@linuxbabu [~]#

I corrected the OS rpm_dist_ver under the config file and the upcp worked fine :

root@linuxbabu [~]# cat /var/cpanel/sysinfo.config | grep rpm_dist_ver
rpm_dist_ver=6
root@linuxbabu [~]#

cPanel : SSHD Rootkit

We have been seeing this SSHD rootkit from time to time with much improved versions 🙂

One could see number of SSH processes on the server with nothing under process details. There is a huge discussion thread at WHT www.webhostingtalk.com/showthread.php?t=1235797

The current one which I have seen today was with keyutils-libs legit version name. However the “Signature” was missing.

root@linuxbabu [/var/log]# ls -la /lib64 | grep libkeyutils
lrwxrwxrwx 1 root root 18 Jun 22 2012 libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 35320 Jun 22 2012 libkeyutils.so.1.3*

root@linuxbabu [/var/log]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 14 Jan 2014 04:56:45 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 59336 License: GPLv2+ and LGPLv2+
Signature : (none)
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.

I matched the size from a clean server and found different in the file size of libkeyutils.so.1.3.

So I reinstalled keyutils-libs replacing the file using yum.

yum reinstall keyutils-libs -y

After reinstall I could see the file size changes and also the Signature shown was correct :

root@linuxbabu [~]# ls -la /lib64/libkeyutils*
lrwxrwxrwx 1 root root 18 Feb 25 01:03 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 10192 Jun 22 2012 /lib64/libkeyutils.so.1.3*
root@linuxbabu [~]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 25 Feb 2014 01:03:47 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 36624 License: GPLv2+ and LGPLv2+
Signature : RSA/SHA1, Sun 24 Jun 2012 05:18:51 PM CDT, Key ID 0946fca2c105b9de
Packager : CentOS BuildSystem
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.
root@linuxbabu [~]#

Hope this helps to disable the rootkit and avoid further damage to the server.

cPanel : /root/ Inode issue!

Today I had an issue on / with shortage of disk inodes on a cPanel dedicated server.

In computing, an inode is a data structure on a traditional Unix-style file system such as UFS. An inode stores basic information about a regular file, directory, or other file system object.

In layman’s term – Every time a file is created or uploaded on a server, an inode is created. Simply say, inode is the count of the number of files on your server may it be a VPS or dedicated server.

The number of inodes used can be checked using below command (example is for /root) :

for i in /root ; do echo $i; find $i -type f | wc -l ;done

For my current issue it was the comet directory which was using up the space / inodes. The comet directory is usually filled on a high mail traffic server.

/root/.cpanel/comet

You can safely clean up the comet directory by running the following command that will remove all files in this directory that have not been accessed in more than three days.

/usr/local/cpanel/bin/purge_dead_comet_files

Example :

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 960992 0 100% /
root@linuxbabu [~]#

root@linuxbabu [~]# /usr/local/cpanel/bin/purge_dead_comet_files
******Cleaning up comet for root...Done
root@linuxbabu [~]#

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 31278 929714 4% /
root@linuxbabu [~]#

This should free your Inodes / Disk space.
🙂

cPanel :: cPHulk error – Error while connecting to MySQL

WHM shows below error for cPHulk Brute Force Protection :

=========
cPHulk Brute Force Protection
Mysql is currently disabled. To enable mysql go to: Service Manager
Once there check Enable and Monitor for mysql. Then at the bottom of the page click save.

=========

Running below command should fix the issue :

/usr/local/cpanel/bin/hulkdsetup

Sample Output :
===========
root@1 [/]# /usr/local/cpanel/bin/hulkdsetup
hulkdsetup: synchronizing database schema

## mysqldiff 0.43
##
## Run on Sat Apr 6 04:05:04 2013
## Options: debug=0, host=localhost
##
## — db: cphulkd (host=localhost)
## +++ file: /usr/local/cpanel/etc/cphulkd_db.sql

CREATE TABLE auths (
SERVER char(128) NOT NULL,
USER char(128) NOT NULL,
PASS char(128) NOT NULL,
PRIMARY KEY (SERVER,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE blacklist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE brutes (
IP char(255) NOT NULL DEFAULT ”,
NOTES text,
BRUTETIME datetime DEFAULT NULL,
EXPTIME datetime DEFAULT NULL,
PRIMARY KEY (IP),
KEY EXPTIME_index (EXPTIME)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE good_logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
LOGINSERVICE char(64) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_LOGINSERVICE_USER_index (LOGINTIME,LOGINSERVICE,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
SERVICE char(64) DEFAULT NULL,
STATUS int(1) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_SERVICE_STATUS_USER_index (LOGINTIME,SERVICE,STATUS,USER),
KEY LOGINTIME_IP_index (LOGINTIME,IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE report (
type char(16) NOT NULL,
service char(16) NOT NULL,
login_service char(16) DEFAULT NULL,
ip char(200) DEFAULT NULL,
user char(100) DEFAULT NULL,
failcount int(11) DEFAULT NULL,
logintime datetime DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE whitelist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
root@1 [/]#

===========

Uninstall CloudLinux :: cPanel server

CloudLinux is a good tool however we had some issues with our test production server and had to remove it. Here are the steps to remove CloudLinux :

CloudLinux Kb has below steps :
https://helpdesk.cloudlinux.com/index.php?/Knowledgebase/Article/View/33/0/how-do-i-convert-back-to-centos

yum remove liblve-devel
cd /usr/local/src/
wget http://repo.cloudlinux.com/cloudlinux/sources/cln/centos2cl
sh centos2cl -c

After executing script from CloudLinux, you need to remove the CL kernels as provided in the output of the script.

eg : (This may be different so watch the output of “sh centos2cl -c”)
rpm -e --nodeps kernel-2.6.32-379.5.1.lve1.1.9.6.1.el6.x86_64 kernel-headers-2.6.32-379.5.1.lve1.1.9.6.1.el6.x86_64 kernel-2.6.32-379.9.1.lve1.1.9.7.1.el6.x86_64 kernel-firmware-2.6.32-379.9.1.lve1.1.9.7.2.el6.noarch kernel-2.6.32-379.9.1.lve1.1.9.7.2.el6.x86_64

Once done install the CentOs kernel :

yum install kernel kernel-headers
yum update

Make sure grub entries are fine and reboot.

Once the server is back execute below command to cleanup CL :

yum remove lve liblve lve-utils lve-stats

Finally run easyapache to rebuild :

/scripts/easyapache --build

You should be fine / free from CloudLinux 🙂

Eximstats too large

If the size of your eximstats database is too large, you can do the following steps to empty it.

Login to mysql server from root.

[root@srv1 ~]# mysql

mysql> use eximstats;
Database changed
mysql> show tables;
+---------------------+
| Tables_in_eximstats |
+---------------------+
| defers |
| failures |
| sends |
| smtp |
+---------------------+
4 rows in set (0.00 sec)

mysql> delete from defers;
Query OK, 0 rows affected (0.52 sec)

mysql> delete from failures;
Query OK, 0 rows affected (0.35 sec)

mysql> delete from sends;
Query OK, 17310 rows affected (0.23 sec)

mysql> delete from smtp;
Query OK, 4678 rows affected (0.06 sec)

mysql> exit
Bye
[root@srv1 ~]#

You can also do this by deleting the above .MYD files from the location /var/lib/mysql/eximstats and restart exim service afterwards.

How to verify matching private key with a certificate

The private key contains a series of numbers. Two of those numbers form the “public key”, the others are part of your “private key”. The “public key” bits are also embedded in your Certificate (we get them from your CSR). To check that the public key in your cert matches the public portion of your private key, you need to view the cert and the key and compare the numbers. To view the Certificate and the key run the commands:

$ openssl x509 -noout -text -in server.crt
$ openssl rsa -noout -text -in server.key

The `modulus’ and the `public exponent’ portions in the key and the Certificate must match. But since the public exponent is usually 65537 and it’s bothering comparing long modulus you can use the following approach:

$ openssl x509 -noout -modulus -in server.crt | openssl md5
$ openssl rsa -noout -modulus -in server.key | openssl md5

And then compare these really shorter numbers. With overwhelming probability they will differ if the keys are different. As a “one-liner”:

$ openssl x509 -noout -modulus -in server.pem | openssl md5 ;\
openssl rsa -noout -modulus -in server.key | openssl md5

And with auto-magic comparison (If more than one hash is displayed, they don’t match):

$ (openssl x509 -noout -modulus -in server.pem | openssl md5 ;\
openssl rsa -noout -modulus -in server.key | openssl md5) | uniq

BTW, if I want to check to which key or certificate a particular CSR belongs you can compute

$ openssl req -noout -modulus -in server.csr | openssl md5

WHMCS Security Advisory

A potential security issue has been discovered whereby it may be possible for a malicious user to inject a specially crafted combination of variables leading to unexpected results. The issue revolves around the Smarty templating system and template related processing.

> Patch Download Link: http://www.whmcs.com/go/21/download

Patch will work for all versions of WHMCS 4.x.

Buy WHMCS cheap at Licensepal.com

How to track which site is using the apache processes.

For apache 1.3, edit your /etc/httpd/conf/httpd.conf and add

ExtendedStatus On <Location /httpd-status> SetHandler server-status Order deny,allow Deny from all Allow from 127.0.0.1 </Location>

 

 

 

just after the code that says “ServerSignature On”. Save, exit, then restart apache. You can access the stats page by going to http://1.2.3.4/httpd-status where 1.2.3.4 is your server’s IP.

If you’re running apache 2.x , then it’s already in the file:
/etc/httpd/conf/extra/httpd.conf
Change the “Allow from” lines to include your IP.

Under cPanel servers the code is already present but you need to uncomment it or just add the code :

<Location /whm-server-status> 
SetHandler server-status 
Order deny,allow 
Deny from all 
Allow from 127.0.0.1
</Location>