So, since we were having trouble with the hardware RAID, we decided to look for alternatives. The ideal solution would aggregate all the storage across 4 servers into one volume. So initially we considered using sharing each of the physical disks via iSCSI, then having some single-head initiator to access them all. The initiator would the be responsible for some type of software RAID, LVM and sharing the NFS mount point. Connectivity and data redundancy are big issues, and I was worried with this method, if one of the servers drops out a lot of data may be lost or corrupted, plus the head may be a bottleneck.
Instead, we decided that each server would host its own NFS share, so we just needed a way to combine all the 3TB physical disks into one large volume. I thought first about a software RAID+ext4, but instead decided to try ZFS. It seemed to have the redundancy we were looking for, plus some other interesting features, such as taking care of NFS from ZFS.
For ZFS we decided to try both Ubuntu and FreeNAS (BSD) to figure out 1) which is faster and 2) which is easier to setup and administer.
After installing Ubuntu 12.04, and following these instructions with a bit of help from here, we had terabytes of aggregate storage accessible over NFS in about 15 minutes. Pretty easy.
Then we started trying to image to the share. From a USB device to the server using Guymager, we could image about 11-13MB/sec. After 30 to 40 seconds of imaging, all the drives in the zpool would start flashing like crazy, and imaging would drop to around 3MB/sec. Faster initial speeds, but similar degradation happened with eSATA/IDE drives on the acquisition machine.
 I believe part of the problem was with NFS having the 'sync' flag set on the server-side. I had less degradation and slightly faster performance with 'async'.
So, for large files ZFS was quite slow. This did NOT seem to be the case when writing from the server to the local zpool, which leads me to think it might be partially a problem with NFS (which zfs handles itself with the 'zfs set sharenfs=on ... ' command). I think the initial high speed is because of caching. So now I am wondering if ZFS is meant for bursts of smaller reads and writes rather than writing large files.
Looking around for why ZFS seems degrade so much, I found this article, which claims to improve ZFS performance. NOTE: if you make any of the changes from the article, make sure you have a backup of your data. I found that after making some of the tweaks, my pool had been corrupted and had to be re-created. The tweaks did seem to make the server more stable over time (less freezing), but write speeds still degraded.
FreeNAS took much less time than Ubuntu to install, and as long as it can detect your disks, creation of a new pool and sharing the pool takes about 10 clicks of your mouse (from the web interface). ZFS on FreeNAS appears to be less stable that on Ubuntu (we had drives drop out, and a more freezing), but we are getting write speeds of approximately 15MB/sec with less - but some - degradation over time.
To see if the slow write speed is ZFS related, we are currently building a software array, and will just format it with ext4, and create a standard NFS share. Syncing the 8 disks in the array will take hours, but after hopefully we will have much, much better transfer speeds. Ubuntu allows the creation of a software array on install, which makes it very easy to set up.
If you have any experience with ZFS, and know what some of the issues might be, or even just a better set-up for aggregating large disks and getting good performance writing large files, please leave a comment.