Another MySQL daemon already running with the same unix socket.

After upgrading MySQL binaries mysqld will not start at all any more and shows below error :

CT-1977-bash-4.1# service mysqld start
Another MySQL daemon already running with the same unix socket.
Starting mysqld: [FAILED]
CT-1977-bash-4.1#

MySQL service does not shut down gracefully during the OS reboot, leaving the old /var/lib/mysql/mysql.sock such that mysqld will not start up. Some people were able to reproduce this error on a CentOS 6.5 KVM guest virtual system by rebooting the host CentOS 6.5 system. CentOS is supposed to gracefully shut down the guest systems, but this seems to be failing in the case of mysqld.

Confirmed Red Hat Linux 6.5 bug – https://bugzilla.redhat.com/show_bug.cgi?id=1037650

Issue discussion on MySQL bug tracker – http://bugs.mysql.com/bug.php?id=71086

Simple steps to reproduce this issue:

service mysqld start
killall -9 mysqld_safe mysqld
service mysqld start

A quick way to restart MySQL is as below :
Remove socket file and restart mysql:
CT-1977-bash-4.1# ls -la /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Jan 8 20:13 /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# rm /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

A work around for the issue is to modify ‘/etc/init.d/mysqld’ script:

Make a backup copy of the startup script.
cp -p /etc/init.d/mysqld /etc/init.d/mysqld.orig

Edit the file /etc/init.d/mysqld to look as below :
=========================
# if fuser "$socketfile" &>/dev/null ; then
# echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
# action $"Starting $prog: " /bin/false
# return 1

# We check if there is already a process using the socket file,
# since otherwise this init script could report false positive
# result and mysqld_safe would remove the socket file, which
# actually uses a different daemon.
if fuser "$socketfile" &>/dev/null ; then
echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
action $"Starting $prog: " /bin/false
return 1
fi
=========================

Now restart MySQL :
CT-1977-bash-4.1# service mysqld start
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

Virtuozzo : RTNETLINK answers: Operation not supported

Here is a new bug which I faced on virtuozzo 4 and 4.6 for Centos 6 VPS.

VPS fails to add IP during start or networking fails when network is restarted :

-bash-4.1# /etc/init.d/network restart
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: RTNETLINK answers: Operation not supported
Failed to bring up lo.
[FAILED]
Bringing up interface venet0: RTNETLINK answers: Operation not supported
Failed to bring up venet0.
[FAILED]
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
-bash-4.1#

ifconfig shows blank as networking fails to start :

-bash-4.1# ifconfig
-bash-4.1# rpm -q iproute
iproute-2.6.32-31.el6.x86_64
-bash-4.1# cat /etc/redhat-release
CentOS release 6.5 (Final)
-bash-4.1#

Here is the fix, download the rpm’s as per your VPS arch.

for 64-bit systems: http://mirror.centos.org/centos/6.4/os/x86_64/Packages/iproute-2.6.32-23.el6.x86_64.rpm
for 32-bit systems: http://mirror.centos.org/centos/6.4/os/i386/Packages/iproute-2.6.32-23.el6.i686.rpm

Networking is not available in VPS so download it on the Hardware Node :

cd /root/dino
wget http://mirror.centos.org/centos/6.4/os/x86_64/Packages/iproute-2.6.32-23.el6.x86_64.rpm

Copy it to the affected VPS :
cp iproute-2.6.32-23.el6.x86_64.rpm /vz/private/VEID/fs/root/root/
replace VEID with the affected VPS ID.

vzctl enter VEID

-bash-4.1# cd /root/
-bash-4.1# rpm -q iproute
iproute-2.6.32-31.el6.x86_64
-bash-4.1# rpm -e iproute --nodeps
-bash-4.1# rpm -Uvh iproute-2.6.32-23.el6.x86_64.rpm
Preparing... ########################################### [100%]
1:iproute ########################################### [100%]
-bash-4.1# /etc/init.d/network restart
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface venet0: Determining if ip address 127.0.0.1 is already in use for device venet0...
SIOCADDRT: Network is unreachable
SIOCADDRT: Network is unreachable
[ OK ]
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
-bash-4.1#

Now you can see the IP’s responding. Please make sure you check the RPM and OS versions:

Parallels KB : http://kb.parallels.com/en/118992

UPDATE :

to avoid updates due to yum append iproute* to exclude line in file /etc/yum.conf

cPanel : /root/ Inode issue!

Today I had an issue on / with shortage of disk inodes on a cPanel dedicated server.

In computing, an inode is a data structure on a traditional Unix-style file system such as UFS. An inode stores basic information about a regular file, directory, or other file system object.

In layman’s term – Every time a file is created or uploaded on a server, an inode is created. Simply say, inode is the count of the number of files on your server may it be a VPS or dedicated server.

The number of inodes used can be checked using below command (example is for /root) :

for i in /root ; do echo $i; find $i -type f | wc -l ;done

For my current issue it was the comet directory which was using up the space / inodes. The comet directory is usually filled on a high mail traffic server.

/root/.cpanel/comet

You can safely clean up the comet directory by running the following command that will remove all files in this directory that have not been accessed in more than three days.

/usr/local/cpanel/bin/purge_dead_comet_files

Example :

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 960992 0 100% /
root@linuxbabu [~]#

root@linuxbabu [~]# /usr/local/cpanel/bin/purge_dead_comet_files
******Cleaning up comet for root...Done
root@linuxbabu [~]#

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 31278 929714 4% /
root@linuxbabu [~]#

This should free your Inodes / Disk space.
🙂

wp-cron.php – High CPU usage

What is wp-cron.php ?

This file is a PHP script which runs all the automated tasks that let WordPress do all it’s wonderful tricks. Some examples include:

 

  • Posting content when it is scheduled to be posted at specific times
  • Check all pending comments for spam (if you have plugins like Akismet running)
  • Send emails (i.e. if you have the option enabled where you get emailed whenever a comment is posted, this script handles the email)

Basically wp-cron.php is the automatic part of WordPress.

WP-Cron.php is a very common cause of high CPU loads. Disabling WP-Cron can drastically reduce CPU-load and prevent the chances your account is suspended due to exceeding your resources.

You can disable WP-Cron by editing your wp-config.php and adding the following line;

define('DISABLE_WP_CRON', 'true');

Create a cron job and run wp-cron.php every hour or two using the following command:

wget -O /dev/null http://www.example.com/wp-cron.php?doing_wp_cron

OR (if wget is disabled) :

cd /home/cpanel_user/public_html; php -q wp-cron.php

🙂

cPanel :: cPHulk error – Error while connecting to MySQL

WHM shows below error for cPHulk Brute Force Protection :

=========
cPHulk Brute Force Protection
Mysql is currently disabled. To enable mysql go to: Service Manager
Once there check Enable and Monitor for mysql. Then at the bottom of the page click save.

=========

Running below command should fix the issue :

/usr/local/cpanel/bin/hulkdsetup

Sample Output :
===========
root@1 [/]# /usr/local/cpanel/bin/hulkdsetup
hulkdsetup: synchronizing database schema

## mysqldiff 0.43
##
## Run on Sat Apr 6 04:05:04 2013
## Options: debug=0, host=localhost
##
## — db: cphulkd (host=localhost)
## +++ file: /usr/local/cpanel/etc/cphulkd_db.sql

CREATE TABLE auths (
SERVER char(128) NOT NULL,
USER char(128) NOT NULL,
PASS char(128) NOT NULL,
PRIMARY KEY (SERVER,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE blacklist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE brutes (
IP char(255) NOT NULL DEFAULT ”,
NOTES text,
BRUTETIME datetime DEFAULT NULL,
EXPTIME datetime DEFAULT NULL,
PRIMARY KEY (IP),
KEY EXPTIME_index (EXPTIME)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE good_logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
LOGINSERVICE char(64) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_LOGINSERVICE_USER_index (LOGINTIME,LOGINSERVICE,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
SERVICE char(64) DEFAULT NULL,
STATUS int(1) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_SERVICE_STATUS_USER_index (LOGINTIME,SERVICE,STATUS,USER),
KEY LOGINTIME_IP_index (LOGINTIME,IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE report (
type char(16) NOT NULL,
service char(16) NOT NULL,
login_service char(16) DEFAULT NULL,
ip char(200) DEFAULT NULL,
user char(100) DEFAULT NULL,
failcount int(11) DEFAULT NULL,
logintime datetime DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE whitelist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
root@1 [/]#

===========

Virtuozzo 4.7 Install Error :: lspci

Virtuozzo 4.7 install show Hardware check errors as below :

=========
[general] error getting information about file /sbin/lspci (No s|
=========

The error is due to missing package pciutils which is not included in CentOS 6 minimal install.

Install pciutils using yum on CentOS server :

yum install pciutils

Once done you can start the install without issues.

Uninstall CloudLinux :: cPanel server

CloudLinux is a good tool however we had some issues with our test production server and had to remove it. Here are the steps to remove CloudLinux :

CloudLinux Kb has below steps :
https://helpdesk.cloudlinux.com/index.php?/Knowledgebase/Article/View/33/0/how-do-i-convert-back-to-centos

yum remove liblve-devel
cd /usr/local/src/
wget http://repo.cloudlinux.com/cloudlinux/sources/cln/centos2cl
sh centos2cl -c

After executing script from CloudLinux, you need to remove the CL kernels as provided in the output of the script.

eg : (This may be different so watch the output of “sh centos2cl -c”)
rpm -e --nodeps kernel-2.6.32-379.5.1.lve1.1.9.6.1.el6.x86_64 kernel-headers-2.6.32-379.5.1.lve1.1.9.6.1.el6.x86_64 kernel-2.6.32-379.9.1.lve1.1.9.7.1.el6.x86_64 kernel-firmware-2.6.32-379.9.1.lve1.1.9.7.2.el6.noarch kernel-2.6.32-379.9.1.lve1.1.9.7.2.el6.x86_64

Once done install the CentOs kernel :

yum install kernel kernel-headers
yum update

Make sure grub entries are fine and reboot.

Once the server is back execute below command to cleanup CL :

yum remove lve liblve lve-utils lve-stats

Finally run easyapache to rebuild :

/scripts/easyapache --build

You should be fine / free from CloudLinux 🙂

CloudLinux Uninstall : Easyapache error : configure: error: C preprocessor “/lib/cpp” fails sanity check

I was testing CloudLinux on a production server however it turned out bad and had to remove it. After removing the kernel and reboot, last step was to run easyapache which gave errors as below :

==========
checking how to run the C preprocessor... /lib/cpp
configure: error: in `/home/cpeasyapache/src/httpd-2.2.23/srclib/apr':
configure: error: C preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
configure failed for srclib/apr
==========

Further checking i found that the kernel-headers package was missing which was installed using command :

=============
yum install kernel-headers-$(uname -r)
=============

After installing kernel headers easyapache executed fine and the server was free from CloudLinux. 🙂

Directadmin :: httpd error Invalid command ‘php_admin_flag’

On Directadmin server httpd fails to start with below error :

# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: Syntax error on line 47 of /usr/local/directadmin/data/users/linuxbabu/httpd.conf:
Invalid command 'php_admin_flag', perhaps misspelled or defined by a module not included in the server configuration

Solution :

Execute below commands to rebuild php and http config files.

./build all d
./build rewrite_confs

Eximstats too large

If the size of your eximstats database is too large, you can do the following steps to empty it.

Login to mysql server from root.

[root@srv1 ~]# mysql

mysql> use eximstats;
Database changed
mysql> show tables;
+---------------------+
| Tables_in_eximstats |
+---------------------+
| defers |
| failures |
| sends |
| smtp |
+---------------------+
4 rows in set (0.00 sec)

mysql> delete from defers;
Query OK, 0 rows affected (0.52 sec)

mysql> delete from failures;
Query OK, 0 rows affected (0.35 sec)

mysql> delete from sends;
Query OK, 17310 rows affected (0.23 sec)

mysql> delete from smtp;
Query OK, 4678 rows affected (0.06 sec)

mysql> exit
Bye
[root@srv1 ~]#

You can also do this by deleting the above .MYD files from the location /var/lib/mysql/eximstats and restart exim service afterwards.