Starting MySQL. ERROR! The server quit without updating PID file - how we fixed it.
Our MySQL databases are backed up frequently, and I've arranged for our server to email me in the event of a failure. From our /bin/bash script which is run by
crontab:
/usr/local/mysql/bin/mysqldump -ufgw -psummat -h127.0.0.1 fgw > /home/backups/$HOUR/fgw.sql
if [ "$?" -eq 0 ]; then
echo "Success $HOUR fgw" > /dev/null
else
echo "Mysqldump encountered a problem $HOUR fgw" | mail -s "FGW BAD dump" graham@wellho.net
fi
I got an email a couple of hours ago ... and it turned out that the MySQL server had halted. Not sure why, which frightens me, but I'm working from a cruise boat with a poor connection and the important thing was to fix it. Trying to restart:
[root@Server51076 init.d]# ./mysql.server start
Starting MySQL. ERROR! The server quit without updating PID file (/usr/local/mysql/data/Server51076.uk2net.com.pid).
[root@Server51076 init.d]#
A quick trip to Google took me on to a couple of articles - which described different problems, or OS Specific things. Several referred to "Google Hell" on this subject, so let me add my own 2 cents worth in case you've arrived here
from Google Hell!
It appears that what's happened is that the MySQL daemon has stopped catastrophically for some reason. It's confused into thinkig it should be referring to a PID file which didn't exist, and because it's crashed, it's refusing to restart. And it's not got any process to tidy us if I try to run server stop scripts. Thus the irritation above.
My solution to get running ...
Ensure that
/etc/my.cnf contains:
[mysqld]
innodb_force_recovery = 1
And restart the server as this will overstep the problem above, and also recover the databases; error recovery levels of 1 or 2 are likely to let you recover (almost) everything we're told, and I have the previous good backup anyway, so I can restore if I need to.
Then stop the server once it's forced the recovery, remove the innodb_force_recovery assignment, and restart it. And - well - it worked for me. I'm monitoring carefully, and posting this to my blog partly as a test and partly in the hope it may provide a helpful clue to others in the future.
(written 2015-05-06)
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
A167 - Web Application Deployment - Shell Programming (bash) [63] Almost like old times - (2004-09-26)
[64] Shell Script for CGI on the web - (2004-09-26)
[749] Cottage industry or production line data handling methods - (2006-06-07)
[827] No news is good news with Unix and Linux - (2006-08-10)
[1287] Work and play at Well House Manor - Football and Shell Shortcuts - (2007-08-02)
[1345] Perl and Shell coding standards / costs of an IT project - (2007-09-11)
[1468] Lexical v Arithemetic testing, Bash and Perl - (2007-12-11)
[1527] Selecting file names in a shell - one word or another - (2008-02-02)
[1904] Ruby, Perl, Linux, MySQL - some training notes - (2008-11-23)
[3791] The Kernel, Shells and Daemons. Greek Gods in computing - (2012-07-01)
[4400] Commenting out an echo killed my bash backup script - (2015-01-19)
[4584] Bash ... some new scripts to - handling user input - (2015-11-27)
[4586] Extending your bash shell with aliases, functions and extra commands - (2015-11-28)
[4587] shell - bash. Writing conditional tests and statements - the options available - (2015-11-28)
A164 - Web Application Deployment - Services and Regular Jobs [544] Repeating tasks with crontab - (2005-12-27)
[907] Browser -> httpd -> Tomcat -> MySQL. Restarting. - (2006-10-28)
[1028] Linux / Unix - process priority and nice - (2007-01-10)
[1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
[1553] Automatic startup and shutdown of Tomcat - (2008-02-24)
[1633] Changing a screen saver from a web page (PHP, Perl, OSX) - (2008-05-06)
[1700] FTP server on Fedora Linux - (2008-07-06)
[1731] Apache httpd, MySQL, PHP - installation procedure - (2008-08-01)
[1733] memcached - overview, installation, example of use in PHP - (2008-08-02)
[1765] Dialects of English and Unix - (2008-08-21)
[1903] daemons - what is running on my Linux server? - (2008-11-23)
[2145] Using the internet to remotely check for power failure at home (PHP) - (2009-04-29)
[2182] What Linux run level am I in? - (2009-05-15)
[3011] What are .pid files? - (2010-10-23)
[3143] On time - (2011-01-23)
[3792] Managing daemons from a terminal session - (2012-07-01)
S153 - Sourcing, Running and Configuring MySQL [192] Current MySQL and PHP paths and upgrades - (2005-01-28)
[334] Symbolic links and hard links - (2005-06-02)
[489] Which MySQL server am I using? - (2005-11-07)
[515] MySQL - an FAQ - (2005-12-03)
[535] MySQL permissions and privileges - (2005-12-20)
[591] Key facts - SQL and MySQL - (2006-02-04)
[1095] Apache httpd , browser, MySQL and MySQL client downloads - (2007-02-28)
[1123] mysqldump and mysqlrestore - (2007-03-30)
[1131] MySQL - Password security (authentication protocol) - (2007-04-02)
[1689] Some sideways thoughts on the news - (2008-06-27)
[1771] More HowTo diagrams - MySQL, Tomcat and Java - (2008-08-24)
[1935] Summary of MySQL installation on a Linux system - (2008-12-11)
[2085] MySQL - licensing issues, even with using the name - (2009-03-16)
[2209] Monitoring and Tuning your MySQL installation - (2009-05-31)
[2426] Which version of MySQL am I running? - (2009-09-26)
[2444] Potted MySQL installation - (2009-10-09)
[2445] Securing MySQL on a production server - (2009-10-09)
[2458] Cant connect to local MySQL server through socket /tmp/mysql.sock - (2009-10-17)
[4390] Checking MySQL database backups have worked (not failed) - (2015-01-10)
[4406] Fixing damaged MySQL tables - Error 1712 and Error 2013 - (2015-01-25)
Some other Articles
Web Server Admin - some of those things that happen, and solutionsWhat and where are the Azores?Election results - what if we had a party list system?Election day, 2015Starting MySQL. ERROR! The server quit without updating PID file - how we fixed it.Newark to New York - the PATH suburban railwayMisusing statistics? - the seedy side of election campaigningTransAtlantic crossing - the starterMoving from mysql to mysqli - simple worked examplePeople in New York