Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
Python and Tcl - public course schedule [here]
Private courses on your site - see [here]
Please ask about maintenance training for Perl, PHP, Lua, etc
Backups by crossover between network centres - setting up automatic scp transfers

Our web servers don't need transaction logging backup systems - emails and secure pages are looked after (and spam trapped) by a separate machine from our two dedicated web servers, which handle a great deal of requests for data reads, but data writes aren't all that many. But of course the data writes / changes / blog articles / forum contribution / calendar changes are happening all the time, and there's a need to be able to restore the machines if something goes dramatically wrong. The question that need to be asked in setting up the backup strategy is "how do we get this back / how much do we loose if the nastiest thing goes wrong at the worst moment?"

Firstly, both machines run regular and quite frequent backups using crontab jobs - with the backups being stored on the same machine, so that we can pull back any data that we need to. Most of these are stamped with the time in the week at which they're taken, so we can step back up to 7 days. These backups have proven very useful when members of forums have done something very silly (like decided to leave in a huff and delete all their posts) ... and we've restored the posts - and thus the integrity of the threads - very quickly indeed. Since of backups are at an SQL and file level, such a restoration doesn't even mean any loss of recent data since the last backup was taken.

But - and this happened a couple of weeks ago - what happens if we have a hard disc failure? "Sorry - there's no point in trying to mount the old disc - there's nothing left on it that can be read...". That's were offsite backups come in and, until not, we've taken such network copies about once a week, when we remember. OK - that's not been very clever, and it takes time - so I've now put into place a data swap scheme between our two machines - on in Germany and the other in England. The question arose how to transfer the backup files cleanly, automatically and securely, and Ive set up a crontab job, twice a week each way, using scp in batch mode. And in order to do that, I had to set up public and private keys between the backup accounts on the two machines. The setup only needs to be done in one direction, as the authorised client scp can both push and pull files. Here are the details ... with keys intentionally changed so that no-one reading this blog can get in!

On the WELL HOUSE MANOR machine - which will be the client that runs the scp command - in Germany

  -bash-3.14b$ ssh-keygen -t dsa
  Generating public/private dsa key pair.
  Enter file in which to save the key (/home/backup/.ssh/id_dsa): my_client_key
  Enter passphrase (empty for no passphrase):
  Enter same passphrase again:
  Your identification has been saved in my_client_key.
  Your public key has been saved in my_client_key.pub.
  The key fingerprint is:
  21:17:81:55:4d:87:aa:e5:0f:34:75:d7:6f:d4:fe:ca backup@p15161732.pureserver.info
  -bash-3.14b$ scp my_client_key.pub backup@www.wellho.net:
  backup@www.wellho.net's password:
  backup@www.wellho.net's password:
  my_client_key.pub 100% |**************************************| 623 00:00

On the WELL HOUSE CONSULTANTS machine - which will be the server - in England

  -bash-4.1$ cat my_client_key.pub >> ~/.ssh/authorized_keys
  -bash-4.1$ chmod 600 !$
  chmod 600 ~/.ssh/authorized_keys
  -bash-4.1$ rm my_client_key.pub

On the client machine in Germany - copy command in the crontab file for user "backup"

  45 7 * * 3 scp -B -r -i /home/backup/my_client_key backup@www.wellho.net:Wed /home/backup/remote
  45 8 * * 3 scp -B -r -i /home/backup/my_client_key /home/backup/local backup@www.wellho.net:remote

repeated TWICE a week, different sources and targets, in case a system fails during transfer

I'm still going to be downloading the occasional full backup to our own HQ and hotel, and also to that mail and secure server I mentioned (which are in California). However, if the world's hit by a disaster big enough to wipe of data centres in London, Koblenz and Fremont, I expect that Lisa and wouldn't be doing very much IT training thereafter!
(written 2013-04-13, updated 2013-04-20)

Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
A162 - Web Application Deployment - Backups and File System Management
  [4481] Extracting data from backups to restore selected rows from MySQL tables - (2015-05-01)
  [4405] Backup procedures - via backup server - (2015-01-24)
  [4400] Commenting out an echo killed my bash backup script - (2015-01-19)
  [4390] Checking MySQL database backups have worked (not failed) - (2015-01-10)
  [4115] More or less back - what happened to our server the other day - (2013-06-14)
  [4056] An overpractical test of our backup strategy! - (2013-03-30)
  [2299] How much space does my directory take - Linux - (2009-07-20)
  [1893] Some Linux and Unix tips - (2008-11-18)
  [1801] Will your backups work if you have to restore them? - (2008-09-18)
  [1765] Dialects of English and Unix - (2008-08-21)
  [1648] The tourists guide to Linux - (2008-05-20)
  [1439] Linux / Unix - layout of operating system files - (2007-11-20)
  [1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
  [1023] Finding public writeable things on your linux file system - (2007-01-06)
  [1013] Copy multiple files - confusing error message from cp - (2006-12-30)
  [754] tar, jar, war, ear, sar files - (2006-06-10)
  [735] Boys will be boys, saved by Ubuntu - (2006-05-27)
  [703] Copying files and preserving ownership - (2006-04-28)
  [593] Finding where the disc space has gone - (2006-02-06)
  [554] What backup is adequate? - (2006-01-04)
  [334] Symbolic links and hard links - (2005-06-02)
  [153] Linux - where to put swap space - (2004-12-16)

Back to
Sessions, forms and validation in CodeIgniter - early examples
Previous and next
Horse's mouth home
Forward to
Apache httpd - a robust, open source web server
Some other Articles
The woman, the television, the bullock and Darlington
MVC and Frameworks - a lesson from first principles in PHP
Handling requests to a forum - the background process
Apache httpd - a robust, open source web server
Backups by crossover between network centres - setting up automatic scp transfers
Sessions, forms and validation in CodeIgniter - early examples
Seamless, integrated IT - we have a long way to go!
CodeIgniter - an excellent PHP framework with an easy start point
Curl and curling from PHP
The highs and lows of customer service - Cheltenham
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2019: 404 The Spa • Melksham, Wiltshire • United Kingdom • SN12 6QL
PH: 01225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/4063_Bac ... sfers.html • PAGE BUILT: Sat May 27 16:49:10 2017 • BUILD SYSTEM: WomanWithCat