6 disk raidz2 or 3 stripe 2 way mirror

[Joe S]

I'm playing around with ZFS and want to figure out the best use of my 6x
300GB SATA drives. The purpose of the drives is to store all of my data at
home (video, photos, music, etc). I'm debating between:

6x 300GB disks in a single raidz2 pool

--or--

2x (3x 300GB disks in a pool) mirrored


I've read up a lot on ZFS, but I can't really figure out which is ideal. I
know that both ways will work but it seems like the 3 disk strip in a 2 way
mirror is the most flexible as far as adding drives. Any help is
appreciated.

Running Solaris 10 with latest patches.

[Ian Collins]

Post by Joe S
I'm playing around with ZFS and want to figure out the best use of my
6x 300GB SATA drives. The purpose of the drives is to store all of my
6x 300GB disks in a single raidz2 pool
--or--
2x (3x 300GB disks in a pool) mirrored

I currently use 4x 2 way mirrors, which for my workload gave the best
performance/capacity ratio. It probably is the easiest configuration to
expand if required.

Ian

[Joe S]

Ok. That helps. I'm trying to wrap my brain around the best way to do this.
I'm going to try to duplicate what you are doing and see how that works out.
I'm going to create 3x 2-way mirrors. I guess I don't really *need* the
raidz at this point. My biggest concern with raidz is getting locked into a
configuration i can't grow out of. I like the idea of adding more 2 way
mirrors to a pool.

Post by Ian Collins

Post by Joe S
I'm playing around with ZFS and want to figure out the best use of my
6x 300GB SATA drives. The purpose of the drives is to store all of my
6x 300GB disks in a single raidz2 pool
--or--
2x (3x 300GB disks in a pool) mirrored

I currently use 4x 2 way mirrors, which for my workload gave the best
performance/capacity ratio. It probably is the easiest configuration to
expand if required.
Ian

[C***@Sun.COM]

Post by Joe S
Ok. That helps. I'm trying to wrap my brain around the best way to do this.
I'm going to try to duplicate what you are doing and see how that works out.
I'm going to create 3x 2-way mirrors. I guess I don't really *need* the
raidz at this point. My biggest concern with raidz is getting locked into a
configuration i can't grow out of. I like the idea of adding more 2 way
mirrors to a pool.

Just as with a mirror you can add a raidz stripe:

zpool add tank raidz c1t0d0 c1t1d0 c1t2d0

Casper

[Marc Bevand]

Post by Joe S
I'm going to create 3x 2-way mirrors. I guess I don't really *need* the
raidz at this point. My biggest concern with raidz is getting locked into
a configuration i can't grow out of. I like the idea of adding more
2 way mirrors to a pool.

The raidz2 option will *not* restrict your possibilities of expansion.

For example, it is perfectly possible to add a mirror to a pool consisting
of single raidz2 vdev.

Plus, compared to 3 2-disk mirrors, a 6-disk raidz2 offers more usable space
and is more reliable.

-marc

[Joe S]

My understanding is that if I create a 6 disk raidz2,

# zpool create tank raidz2 disk0 disk1 disk2 disk3 disk4 disk5

I cannot add more disks to this set. I cannot expand this. I have to destroy
the raidz2 it and recreate it if I want to increase capacity.
Whereas with 2 way mirrors, i can just keep adding pairs of drives, or
replace old 2 way pairs with new larger capacity pairs.

At first this raidz stuff was really simple, until I thought about adding
more drives or replacing the old drives with larger drives in the future.
I'm trying to make this scale in the future, but now it looks like I'm not
understanding raidz.

Post by Marc Bevand

Post by Joe S
I'm going to create 3x 2-way mirrors. I guess I don't really *need* the
raidz at this point. My biggest concern with raidz is getting locked

into

Post by Joe S
a configuration i can't grow out of. I like the idea of adding more
2 way mirrors to a pool.

The raidz2 option will *not* restrict your possibilities of expansion.
For example, it is perfectly possible to add a mirror to a pool consisting
of single raidz2 vdev.
Plus, compared to 3 2-disk mirrors, a 6-disk raidz2 offers more usable space
and is more reliable.
-marc
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[John-Paul Drawneek]

ok if its just storing stuff raidz2 is probably the best use of space.
raidz2 on 5 disk and one spare - this can take 3 disk failing before you lose your data.

The three strip mirror will give you nice performance but from the sounds of it you don't need it.


This message posted from opensolaris.org

[Richard Elling]

Post by Joe S
I'm playing around with ZFS and want to figure out the best use of my 6x
300GB SATA drives. The purpose of the drives is to store all of my data
6x 300GB disks in a single raidz2 pool
--or--
2x (3x 300GB disks in a pool) mirrored
I've read up a lot on ZFS, but I can't really figure out which is ideal.
I know that both ways will work but it seems like the 3 disk strip in a
2 way mirror is the most flexible as far as adding drives. Any help is
appreciated.

This is becoming one of the most frequently asked questions.
The good news is that you have options. The bad news is that
you have options. Deciding which option is best forces you to
prioritize, and different people value different features. I've
put together a series of blogs which you might find useful to
help make such decisions (space vs. RAS vs. performance)
http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1

-- richard

[Joe S]

Ah, so you are the Richard behind those articles I've been mulling over! :-)
You blog posts helped me to realize there was much more forethought required
when setting up my ZFS pool(s).

I'm glad I'm not the only person with this question.

Whatever I decide, I will include in this thread.

Post by Richard Elling

Post by Joe S
I'm playing around with ZFS and want to figure out the best use of my 6x
300GB SATA drives. The purpose of the drives is to store all of my data
6x 300GB disks in a single raidz2 pool
--or--
2x (3x 300GB disks in a pool) mirrored
I've read up a lot on ZFS, but I can't really figure out which is ideal.
I know that both ways will work but it seems like the 3 disk strip in a
2 way mirror is the most flexible as far as adding drives. Any help is
appreciated.

This is becoming one of the most frequently asked questions.
The good news is that you have options. The bad news is that
you have options. Deciding which option is best forces you to
prioritize, and different people value different features. I've
put together a series of blogs which you might find useful to
help make such decisions (space vs. RAS vs. performance)
http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1
-- richard

[Joe S]

I'm going to try 5 disks in raidz2 with 1 hot spare.

I read about this here:

http://prefetch.net/blog/index.php/2007/02/09/using-raidz2-and-hot-spares-on-older-sun-storage-arrays/

I don't have older disks, but they are consumer grade disks, and I've been
bitten by disks going dead before, thus this whole thread.

Post by Joe S
Ah, so you are the Richard behind those articles I've been mulling over! :-)
You blog posts helped me to realize there was much more forethought
required when setting up my ZFS pool(s).
I'm glad I'm not the only person with this question.
Whatever I decide, I will include in this thread.

Post by Joe S

Post by Joe S
I'm playing around with ZFS and want to figure out the best use of my

6x

Post by Joe S
300GB SATA drives. The purpose of the drives is to store all of my

data

Post by Joe S
6x 300GB disks in a single raidz2 pool
--or--
2x (3x 300GB disks in a pool) mirrored
I've read up a lot on ZFS, but I can't really figure out which is

ideal.

Post by Joe S
I know that both ways will work but it seems like the 3 disk strip in

a

Post by Joe S
2 way mirror is the most flexible as far as adding drives. Any help is
appreciated.

This is becoming one of the most frequently asked questions.
The good news is that you have options. The bad news is that
you have options. Deciding which option is best forces you to
prioritize, and different people value different features. I've
put together a series of blogs which you might find useful to
help make such decisions (space vs. RAS vs. performance)
http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1
-- richard

[Joe S]

After getting a few replies on this, I've realized the following:

If I have 3 disks in a raidz vdev and I want to add more storage in the
future, I would need add another 3 disk raidz vdev.
If I have 4 disks in a raidz2 vdev and I want to add more storage in the
future, I would need add another 4 disk raidz2 vdev.
If I have multiple 2 disk mirrors in a pool and I want to add more storage
in the future, I would need to add another 2 disk mirror.

Mirroring seems to be my least favorite option since it uses up so much
space. I could also mix and match the above, but I think I will get confused
later on down the road. ;-)

My goal is to have some sort of fault tolerance for this data, but I know I
will need to grow it someday as drives only last so long and larger drives
will be more affordable and readily available.

Thanks for the replies so far. I want to do this right the first time. I
didn't realize the task of choosing what kind fault tolerance to use in ZFS
would be this difficult! But I am very impressed with how quickly and easily
I can destroy and recreate pools.

Post by Joe S
I'm playing around with ZFS and want to figure out the best use of my 6x
300GB SATA drives. The purpose of the drives is to store all of my data at
6x 300GB disks in a single raidz2 pool
--or--
2x (3x 300GB disks in a pool) mirrored
I've read up a lot on ZFS, but I can't really figure out which is ideal. I
know that both ways will work but it seems like the 3 disk strip in a 2 way
mirror is the most flexible as far as adding drives. Any help is
appreciated.
Running Solaris 10 with latest patches.

[Darren Dunham]

Post by Joe S
If I have 3 disks in a raidz vdev and I want to add more storage in the
future, I would need add another 3 disk raidz vdev.

This is not a constraint that ZFS forces on you.

Post by Joe S
If I have 4 disks in a raidz2 vdev and I want to add more storage in the
future, I would need add another 4 disk raidz2 vdev.
If I have multiple 2 disk mirrors in a pool and I want to add more storage
in the future, I would need to add another 2 disk mirror.

And neither are these.

The configuration of any vdev that you create does not constrain you
with any vdevs you want to add to the pool in the future. You can start
with any of your three choices above and then add any of the other three
to the same pool.

Now some additions may not make sense for what you want to do, but they
are available.

[Brian Hechinger]

Post by Darren Dunham
The configuration of any vdev that you create does not constrain you
with any vdevs you want to add to the pool in the future. You can start
with any of your three choices above and then add any of the other three
to the same pool.

'zpool add' will complain if the number of disks aren't equal, forcing you
to use -f to get it to work. I wonder why this is? Does this have to do
with the way zfs determines what goes where? Does whatever system that is
used work better with similar vdevs?

Post by Darren Dunham
Now some additions may not make sense for what you want to do, but they
are available.

I setup a test zfs sytem at work for some of the other non-solaris savvy
linux admins and it had a 6x36g raidz, a 6x73g raidz and a 143g mirror.
It's not something we pushed hard, so I don't know what the performance
impact would be, but it certainly did work. :)

Which raises the question.....

What are the performance implications of pairing different "types" of
vdevs in a single pool?

-brian

[Darren Dunham]

Post by Brian Hechinger

Post by Darren Dunham
The configuration of any vdev that you create does not constrain you
with any vdevs you want to add to the pool in the future. You can start
with any of your three choices above and then add any of the other three
to the same pool.

'zpool add' will complain if the number of disks aren't equal, forcing you
to use -f to get it to work. I wonder why this is? Does this have to do
with the way zfs determines what goes where? Does whatever system that is
used work better with similar vdevs?

I think it's mainly to keep you from doing something silly without
meaning to.

If you have the same type and columns, then you have the same
availability expectations. If instead you take a 5 disk raidz and add a
2 disk stripe, you're dramatically changing the availability
expectations. So the force is required.

Post by Brian Hechinger

Post by Darren Dunham
Now some additions may not make sense for what you want to do, but they
are available.

I setup a test zfs sytem at work for some of the other non-solaris savvy
linux admins and it had a 6x36g raidz, a 6x73g raidz and a 143g mirror.
It's not something we pushed hard, so I don't know what the performance
impact would be, but it certainly did work. :)
Which raises the question.....
What are the performance implications of pairing different "types" of
vdevs in a single pool?

It's going to try to stripe across them without regard for performance
today. So one especially slow vdev (maybe a single disk?) could slow
the overall performance for a pool that has other faster components.

[Brian Hechinger]

Post by Darren Dunham
I think it's mainly to keep you from doing something silly without
meaning to.

That's certainly a good reason. :)

Post by Darren Dunham
If you have the same type and columns, then you have the same
availability expectations. If instead you take a 5 disk raidz and add a
2 disk stripe, you're dramatically changing the availability
expectations. So the force is required.

Certainly, but adding a mirror doesn't degrade the availability expectations,
so it's a fairly safe addition.

Post by Darren Dunham
It's going to try to stripe across them without regard for performance
today. So one especially slow vdev (maybe a single disk?) could slow
the overall performance for a pool that has other faster components.

Ok, so performance will be based on the slowest vdev for the most part.
I can live with that. ;)

I'm sure this is far from simple, but it would be interesting to see zfs
able to figure out the relative performance characteristics of each of
its vdevs and spread the load accordingly. :)

-brian

[Darren Dunham]

Post by Brian Hechinger

Post by Darren Dunham
If you have the same type and columns, then you have the same
availability expectations. If instead you take a 5 disk raidz and add a
2 disk stripe, you're dramatically changing the availability
expectations. So the force is required.

Certainly, but adding a mirror doesn't degrade the availability expectations,
so it's a fairly safe addition.

Nonetheless, it's different. It does change the cost expectations, so
you could consider that it's a warning for that reason then.

Post by Brian Hechinger
I'm sure this is far from simple, but it would be interesting to see zfs
able to figure out the relative performance characteristics of each of
its vdevs and spread the load accordingly. :)

There are quite a few ideas around that. Simply the ability to
recognize "slow" storage would be a good idea. Then you can make other
decisions (lower it for write priority? temporarily not use it for
writes? Decide it's so bad that you should fail it?....)

[Joe S]

Thanks for all the comments. Very helpful.

I have another question. The six disk raidz2 pool works, but I noticed in
Richard Elling's blog that a raidz/raidz2 pool has the read performance of a
single drive (unless I misread something). What if I create 2x three disk
raidz vdevs and put them in a pool? That would be like RAID 5-0 right? Any
disadvantage to doing this? It looks like I'll get the read performance of 2
disks.


-joe

Post by Joe S
I'm playing around with ZFS and want to figure out the best use of my 6x
300GB SATA drives. The purpose of the drives is to store all of my data at
6x 300GB disks in a single raidz2 pool
--or--
2x (3x 300GB disks in a pool) mirrored
I've read up a lot on ZFS, but I can't really figure out which is ideal. I
know that both ways will work but it seems like the 3 disk strip in a 2 way
mirror is the most flexible as far as adding drives. Any help is
appreciated.
Running Solaris 10 with latest patches.

[Ian Collins]

Post by Joe S
Thanks for all the comments. Very helpful.
I have another question. The six disk raidz2 pool works, but I noticed
in Richard Elling's blog that a raidz/raidz2 pool has the read
performance of a single drive (unless I misread something). What if I
create 2x three disk raidz vdevs and put them in a pool? That would be
like RAID 5-0 right? Any disadvantage to doing this? It looks like
I'll get the read performance of 2 disks.

Try it!

Ian

[Will Murnane]

Post by Joe S
Thanks for all the comments. Very helpful.
I have another question. The six disk raidz2 pool works, but I noticed in
Richard Elling's blog that a raidz/raidz2 pool has the read performance of a
single drive (unless I misread something). What if I create 2x three disk
raidz vdevs and put them in a pool? That would be like RAID 5-0 right? Any
disadvantage to doing this? It looks like I'll get the read performance of 2
disks.

If you do small, random reads, then his numbers hold. If you do
full-stripe reads and writes (which (for example) playing a movie
does) then you need to look elsewhere for your numbers.

The other thing to consider is that for most home uses, single-disk
performance is plenty. Even if you only get performance like that of
one disk, it's overkill even for such demanding-sounding things as
streaming four HD movies at once*. Sequential reads are plenty fast.
It's generally only when things get small and/or random that disk I/O
becomes a bottleneck.

Will

*Blu-ray maximum mux rate is 54 megabits per second. That's about 7
MB/s. HD-DVD is lower. x264 is lower still, generally. Disks are,
generally speaking, fast enough to do a couple streaming reads at that
speed.