[wellylug] High load averages but no apparent cause

[David Harrison]

I thought it was I/O bound too as it is running a software RAID5 array.
I would like it to be better, but the client can't afford hardware upgrades
right now.

Does 6% IO wait time (from vmstat) constitute really bad disk performance?
I've got systems with much higher wait times that have far lower loads.

Here's the header output from top (the two active processes are registering
1% cpu load each):

top - 17:05:54 up 2 days, 20:25,  1 user,  load average: 2.74, 1.17, 0.74
Tasks:  71 total,   2 running,  69 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 14.3%id, 85.4%wa,  0.0%hi,  0.1%si,
 0.0%st
Mem:   2074112k total,  2020768k used,    53344k free,    13128k buffers
Swap:  3903608k total,      828k used,  3902780k free,  1770188k cached


Here's the vmstat output:
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
wa
 0  0    828  53416  13228 1770304    0    0    12    22    0   24  0  4 90
 6

And finally iostat
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                  0.28     0.00        4.47      5.73      0.00    89.52

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4.12        22.55        56.06    5558094   13814570
sdb               3.79        22.91        56.63    5646538   13955010
sdc               4.06        23.30        56.80    5742544   13997936
md0               0.00         0.02         0.00       3756         10
md1               0.00         0.01         0.01       1592       2176
md2               1.26         2.60         9.28     640370    2287456
md3               9.40        46.87        58.95   11551194   14527312



On Mon, Mar 22, 2010 at 5:02 PM, Daniel Reurich <daniel at centurion.net.nz>wrote:

> On Mon, 2010-03-22 at 16:49 +1300, David Harrison wrote:
> > Hi,
> > Has anyone experienced high-load averages but haven't been able to see
> > processes that are causing it?
> >
> >
> > I've got an Ubuntu Server 9.10 instance who's load average ranges
> > between 1.0 and 3.0 for most of the day, yet tools like top and iostat
> > don't reveal any issues.
> > i.e. The load averages can be up around 1.5 whilst the maximum process
> > viewed in top is sitting at 5% of the CPU.
> >
> >
> > Anyone know of any other good tools for identifying the cause of
> > server load if the obvious ones fail?
> >
> What's the wait state like (in top it's the %wa value).
>
> Chances are that you have some serious I/O blocking going on which could
> be a slow or failing harddisk or something like that.
>
>
>
> --
> Daniel Reurich.
>
> Centurion Computer Technology (2005) Ltd
> Mobile 021 797 722
>
>
>
>
> --
> Wellington Linux Users Group Mailing List: wellylug at lists.wellylug.org.nz
> To Leave:  http://lists.wellylug.org.nz/mailman/listinfo/wellylug
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wellylug.org.nz/pipermail/wellylug/attachments/20100322/d1d13538/attachment.htm