[wellylug] XFS and JFS

[Daniel Pittman]

E Chalaron <e.chalaron at xtra.co.nz> writes:

>> 40000 files per directory? I assume that there is no other way of doing 
>> it? My gut feel is that the filesystem might have an effect, but it will 
>> be small.
>
> Unfortunately not, I cant go around that. They are relatively big files 
> (5 M each) of images to be transformed in (usually) a 150 G video file.
> Each individual file is written on the HD at a rate of 7 to 8 per 
> seconds through firewire.
>
> For whatever reason when I reach the 30000 I start to notice that the OS 
> is struggling a bit despite a Raid 0 (2x 500 G) set up.

Mmmm.  At least some file systems under Linux have a 32K limit on the
number of files in the directory.

> Was wondering if changing to XFS could fix it or just reformatting md0 
> with ext3 big tree option (as suggested previously)

B+ tree, not big tree, and you are likely to see better performance in
your use case with XFS than with ext3; your file use is one of their
original target markets, and the file system works very well with most
video applications.

Be aware, though, that unless you are running an extremely recent kernel
(2.6.24, the latest release, in fact) then there are risks that files
make have their content replaced with NULL after a file system crash in
some circumstances.[1]

I presume you have a UPS setup if you are doing video work, but be
careful that you can cope with the risks if you depend on the journal
recovery of the file system only.

See the XFS FAQ for more details, if you don't already know.

Regards,
        Daniel

Footnotes: 
[1]  Specifically, files that are truncated longer than their original
     length before being written, and a few less common circumstances.

-- 
Daniel Pittman <daniel at cybersource.com.au>           Phone: 03 9428 6922
1/130-132 Stawell St, Richmond              Web: http://www.cyber.com.au
Cybersource: Australia's Leading Linux and Open Source Solutions Company