Discussion:
pool layout vs resilver times
(too old to reply)
Russ Poyner
2013-01-05 17:42:31 UTC
Permalink
Raw Message
I'm configuring a box with 24x 3Tb consumer SATA drives, and wondering
about the best way to configure the pool. The customer wants capacity on
the cheap, and I want something I can service without sweating too much
about data loss. Due to capacity concerns raid 10 is probably out, which
leaves various raidz choices

1. A stripe of four 6 disk raidz2

2. A stripe of two 11 disk raidz3 with 2 hot spares.

Other, better ideas?

My questions are

A. How long will resilvering take with these layouts when the disks
start dying?

B. Should I prefer hot spares or additional parity drives, and why?

The box is a supermicro with 36 bays controlled through a single LSI
9211-8i. There is a separate intel 320 ssd for the OS. The purpose is to
backup data from the customer's windows workstations. I'm leaning toward
using BackupPC for the backups since it seems to combine good efficiency
with a fairly customer-friendly web interface.

I'm running FreeBSD 9, after having failed to get the plugin jail
working in FreeNAS, also for personal reasons I find csh easier to use
than the FreeNAS web interface. My impression is that FreeBSD combines a
mature OS with the second oldest/best (after Illumos) free
implementation of zfs.

Thanks in advance
Russ Poyner
Richard Elling
2013-01-06 05:11:21 UTC
Permalink
Raw Message
I'm configuring a box with 24x 3Tb consumer SATA drives, and wondering about the best way to configure the pool. The customer wants capacity on the cheap, and I want something I can service without sweating too much about data loss. Due to capacity concerns raid 10 is probably out, which leaves various raidz choices
You should consider space, data dependability as measured by Mean Time to Data
Loss (MTTDL), and performance.

For MTTDL[1] model, let's use 700k hours MTBF for the disks and 168 hours for
recovery (48 hours logistical + 120 hours resilver of a full disk)

For performance, lets hope for 7,200 rpms and about 80 IOPS for small, random reads
with 100% cache miss.
1. A stripe of four 6 disk raidz2
Option 1
space ~= 4 * (6 - 2) * 3TB = 48 TB
MTTDL[1] = 8.38e+5 years or 0.000119% Annualized Failure Rate (AFR)
small, random read performance, best of worst case = 4 * (6/4) * 80 IOPS = 480 IOPS
2. A stripe of two 11 disk raidz3 with 2 hot spares.
Option 2
space ~= 2 * (11 - 3) * 3TB = 48 TB
MTTDL[1] = 3.62e+7 years or 0.000003% AFR
small, random read performance, best of worst case = 2 * (11/8) * 80 IOPS = 220 IOPS

Option 2a (no hot spares)
space ~= 2 * (12 - 3) * 3TB = 54 TB
MTTDL[2] = 1.90e+7 years or 0.000005% AFR
small, random read performance, best of worst case = 2 * (12/9) * 80 IOPS = 213 IOPS
Other, better ideas?
There are thousands of permutations you could consider :-)
For 24-bay systems with double parity or better, we also see a 3x8-disk as a
common configuration. Offhand, I'd say we see more 4x6-disk and 3x8-disk
configs than any configs with more than 10 disks per set.
My questions are
A. How long will resilvering take with these layouts when the disks start dying?
It depends on the concurrent workload. By default resilvers are throttled and give
way to other workload. In general, for double or triple parity RAID, you don't need
to worry too much on a per-disk basis. The conditions you need to worry about are
where the failure cause is common to all disks, such as a controller, fans, cabling,
or power because they are more likely than a triple failure of disks (as clearly shown
by the MTTDL[1] model results above)
B. Should I prefer hot spares or additional parity drives, and why?
In general, addional parity is better than hot spares. You get more performance
and better data dependability.
The box is a supermicro with 36 bays controlled through a single LSI 9211-8i. There is a separate intel 320 ssd for the OS. The purpose is to backup data from the customer's windows workstations. I'm leaning toward using BackupPC for the backups since it seems to combine good efficiency with a fairly customer-friendly web interface.
Sounds like a good plan.
-- richard
I'm running FreeBSD 9, after having failed to get the plugin jail working in FreeNAS, also for personal reasons I find csh easier to use than the FreeNAS web interface. My impression is that FreeBSD combines a mature OS with the second oldest/best (after Illumos) free implementation of zfs.
Thanks in advance
Russ Poyner
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--

***@RichardElling.com
+1-760-896-4422
Dave Pooser
2013-01-06 06:21:05 UTC
Permalink
Raw Message
Post by Russ Poyner
I'm configuring a box with 24x 3Tb consumer SATA drives
<snip>
Post by Russ Poyner
The box is a supermicro with 36 bays controlled through a single LSI
9211-8i.
My recollection is that it's far from best practice to have SATA drives
connected to a SAS expander; better to either use SAS drives or use one of
the Supermicro chassis designs that doesn't use expanders in their
backplanes and control the drives with multiple LSI cards. If you've
already purchased the configuration as described, you may be a little
stuck with it, but my understanding is that the combination of SATA drives
and SAS expanders is a large economy-sized bucket of pain.
--
Dave Pooser
Manager of Information Services
Alford Media http://www.alfordmedia.com
Loading...