Hi,<div>I ran the smartctl tests (both short and long) on all three physical drives overnight.</div><div>It showed all drives were working 100% correctly.</div><div><br></div><div>Overnight I also ran a number of read/write tests and monitored the i/o status in vmstat and iostat.</div>
<div><br></div><div>It seems like performance falls through the floor as soon as the physical memory on the server is exhausted.</div><div><br></div><div>The issue I am experiencing seems to be very similar to the issue which is documented here:</div>
<div><a href="http://notemagnet.blogspot.com/2008/08/linux-write-cache-mystery.html">http://notemagnet.blogspot.com/2008/08/linux-write-cache-mystery.html</a></div><div><br></div><div>I've checked the kernel parameters that are mentioned in this article (dirty_ratio and dirty_background_ratio) and they are the values that are recommended.</div>
<div><br></div><div>Putting more RAM in the machine will certainly forestall the issue, but beyond that it maybe a case of trying RAID1 instead of RAID5.</div><div><br></div><div><br>David</div><div><br></div><div><br><br>
<div class="gmail_quote">On Tue, Mar 23, 2010 at 9:42 AM, David Harrison <span dir="ltr"><<a href="mailto:david.harrison@stress-free.co.nz">david.harrison@stress-free.co.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Cheers Daniel.<div>Will do tonight out of hours.</div><div><br></div><div>I hope it isn't one of the drives, they are brand new and the servers in Auckland whilst I am here in Wellington...</div><div><br></div><font color="#888888"><div>
<br>
David</div></font><div><div></div><div class="h5"><div><br><br><div class="gmail_quote">On Tue, Mar 23, 2010 at 9:20 AM, Daniel Reurich <span dir="ltr"><<a href="mailto:daniel@centurion.net.nz" target="_blank">daniel@centurion.net.nz</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On Tue, 2010-03-23 at 08:50 +1300, David Harrison wrote:<br>
> Thanks Daniel that switch for vmstat is very handy and I'd completely<br>
> missed the io wait value in top.<br>
> Googling based on your comments also brought up this page which is<br>
> very handy:<br>
> <a href="http://strugglers.net/wiki/Linux_performance_tuning" target="_blank">http://strugglers.net/wiki/Linux_performance_tuning</a><br>
><br>
><br>
> The problem is certainly is looking like an intermittent I/O issue.<br>
><br>
</div>Probably caused by a disk issue - seriously. I have seen this before.<br>
<div>><br>
> Has anyone experience with the performance boost of a dedicated PCI-X<br>
> SATA controller for software RAID?<br>
<br>
><br>
> The server in question is a bog-standard HP ML110.<br>
> It isn't up to their needs, but it was a recent purchase by the<br>
> previous IT guys, so I'm afraid it is staying.<br>
><br>
</div>I don't think it should be an issue.<br>
<div>><br>
> For practical reasons I want to keep the software RAID-5 (3x1TB<br>
> drives), but would putting some (or all) of these disks onto a<br>
> dedicated controller alleviate the I/O issue?<br>
><br>
</div>I'd say it would provide none to minimal gain given your load stats if<br>
you are using software raid, and a small gain if you get hardware raid.<br>
<div>><br>
> i.e. Is it worth recommending a $400 PCI-X SATA controller for the<br>
> box, or is that money better left on the table for a new server<br>
> (ML310/330) in twelve months time?<br>
> (My concern being that the card goes in and the problem remains the<br>
> same.)<br>
<br>
</div>Save the money unless you can be sure it's the built in controller (I<br>
doubt it is).<br>
<br>
I think your next port of call is to d a SMART check on your drives.<br>
Install smartmontools and do a smartctl -son /dev/sdX for each drive,<br>
and report that back if you don't understand the results.<br>
<br>
Regards,<br>
Daniel.<br>
<font color="#888888"><br>
<br>
<br>
<br>
<br>
--<br>
</font><div><div></div><div>Wellington Linux Users Group Mailing List: <a href="mailto:wellylug@lists.wellylug.org.nz" target="_blank">wellylug@lists.wellylug.org.nz</a><br>
To Leave: <a href="http://lists.wellylug.org.nz/mailman/listinfo/wellylug" target="_blank">http://lists.wellylug.org.nz/mailman/listinfo/wellylug</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>