[wellylug] High load averages but no apparent cause

David Harrison david.harrison at stress-free.co.nz
Wed Mar 24 08:27:57 NZDT 2010


Hi,
I ran the smartctl tests (both short and long) on all three physical drives
overnight.
It showed all drives were working 100% correctly.

Overnight I also ran a number of read/write tests and monitored the i/o
status in vmstat and iostat.

It seems like performance falls through the floor as soon as the physical
memory on the server is exhausted.

The issue I am experiencing seems to be very similar to the issue which is
documented here:
http://notemagnet.blogspot.com/2008/08/linux-write-cache-mystery.html

I've checked the kernel parameters that are mentioned in this article
(dirty_ratio and dirty_background_ratio) and they are the values that are
recommended.

Putting more RAM in the machine will certainly forestall the issue, but
beyond that it maybe a case of trying RAID1 instead of RAID5.


David



On Tue, Mar 23, 2010 at 9:42 AM, David Harrison <
david.harrison at stress-free.co.nz> wrote:

> Cheers Daniel.
> Will do tonight out of hours.
>
> I hope it isn't one of the drives, they are brand new and the servers in
> Auckland whilst I am here in Wellington...
>
>
> David
>
>
> On Tue, Mar 23, 2010 at 9:20 AM, Daniel Reurich <daniel at centurion.net.nz>wrote:
>
>> On Tue, 2010-03-23 at 08:50 +1300, David Harrison wrote:
>> > Thanks Daniel that switch for vmstat is very handy and I'd completely
>> > missed the io wait value in top.
>> > Googling based on your comments also brought up this page which is
>> > very handy:
>> > http://strugglers.net/wiki/Linux_performance_tuning
>> >
>> >
>> > The problem is certainly is looking like an intermittent I/O issue.
>> >
>> Probably caused by a disk issue - seriously.  I have seen this before.
>> >
>> > Has anyone experience with the performance boost of a dedicated PCI-X
>> > SATA controller for software RAID?
>>
>> >
>> > The server in question is a bog-standard HP ML110.
>> > It isn't up to their needs, but it was a recent purchase by the
>> > previous IT guys, so I'm afraid it is staying.
>> >
>> I don't think it should be an issue.
>> >
>> > For practical reasons I want to keep the software RAID-5 (3x1TB
>> > drives), but would putting some (or all) of these disks onto a
>> > dedicated controller alleviate the I/O issue?
>> >
>> I'd say it would provide none to minimal gain given your load stats if
>> you are using software raid, and a small gain if you get hardware raid.
>> >
>> > i.e. Is it worth recommending a $400 PCI-X SATA controller for the
>> > box, or is that money better left on the table for a new server
>> > (ML310/330) in twelve months time?
>> > (My concern being that the card goes in and the problem remains the
>> > same.)
>>
>> Save the money unless you can be sure it's the built in controller (I
>> doubt it is).
>>
>> I think your next port of call is to d a SMART check on your drives.
>> Install smartmontools and do a smartctl -son /dev/sdX for each drive,
>> and report that back if you don't understand the results.
>>
>> Regards,
>>        Daniel.
>>
>>
>>
>>
>>
>> --
>> Wellington Linux Users Group Mailing List: wellylug at lists.wellylug.org.nz
>> To Leave:  http://lists.wellylug.org.nz/mailman/listinfo/wellylug
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wellylug.org.nz/pipermail/wellylug/attachments/20100324/2b539d7d/attachment.htm 


More information about the wellylug mailing list