« Tuning httpd / the supermarket checkout comparison | Main | Web Site Loading - experiences and some solutions shared »
February 26, 2009
Effect on server when memory runs out and swapping starts
Here's an interesting example from a web server showing what happens when real memory runs out and the system starts swapping ... it can be something of a downwards spiral in terms of performance, as the majority of the resources are essentially tied up by the swapping / keeping going at all once memory is full, and a hard to recover backlog of processes can build up.
First, an example where the system is running in memory:
And this is what can happen with a slightly higher arrival rate, and / or a slightly slower processing rate such that a queue builds up:
You'll notice by comparing these two (look at the purple highlights):
1. The load average has rocketed from 0.23 jobs waiting (average in last minute) to 22.59 jobs.
2. The cpu utilisation remains low - under 10% - for user processes
3. The cpu idle time has dropped from nearly 90% to zero, with all the extra resource being burned up by device wait time - 93.6% no less.
4. Swap space has only increased slightly in its use, but every process in this machine is active and the whole thing hsa started thrashing which means a queue has built up ...
5. There's been a rise from 115 to 191 total tasks (remember that the majority of the initial 115 will be operating system background processes, so that this is a truly massive increase)
6. The process table itself is showing details of just how / where the queue has built up.
See here for some tuning information for httpd if it's the web server that's consuming all the space (there are MySQL notes there too!)
Posted by gje at February 26, 2009 08:05 AM