vdev failure -> pool loss ?

[Simon Breden]

I would just like to confirm or not whether a vdev failure would lead to failure of the whole pool or not.

For example, if I created a pool from two RAID-Z2 vdevs, and three drives fail within the first vdev, is all the data within the whole pool unrecoverable?

[Ian Collins]

Post by Simon Breden
I would just like to confirm or not whether a vdev failure would lead to failure of the whole pool or not.
For example, if I created a pool from two RAID-Z2 vdevs, and three drives fail within the first vdev, is all the data within the whole pool unrecoverable?

Yes. It would be the same a losing a drive in a stripe of two drives.

[Simon Breden]

OK, thanks Ian.

Another example:

Would you lose all pool data if you had two vdevs: (1) a RAID-Z2 vdev and (2) a two drive mirror vdev, and three drives in the RAID-Z2 vdev failed?

[Freddie Cash]

Post by Simon Breden
OK, thanks Ian.
Would you lose all pool data if you had two vdevs: (1) a RAID-Z2 vdev and (2) a two drive mirror vdev, and three drives in the RAID-Z2 vdev failed?

If you lose 1 vdev, you lose the pool.

Doesn't matter what the redundancy level used in the vdev, if you lose
more drives than the vdev can handle, you lose the whole pool.

An easy way to remember this:
- think of vdevs as harddrives
- think of the pool as a RAID-0 stripeset

Thus, if 1 drive dies, everything in the RAID-0 is lost.

Similar for the pool.

[Simon Breden]

OK, thanks Freddie, that's pretty clear.

Cheers,
Simon

[Edward Ned Harvey]

Post by Freddie Cash
If you lose 1 vdev, you lose the pool.

As long as 1 vdev is striped and not mirrored, that's true.
You can only afford to lose a vdev, if your vdev itself is mirrored.

You could, for example, create 2 vdev's of raidz2, and instead of striping
them together, mirror them. Then you could lose one of the raidz2's, and
the other half of the mirror is still up.

[Peter Tribble]

Post by Edward Ned Harvey

Post by Freddie Cash
If you lose 1 vdev, you lose the pool.

As long as 1 vdev is striped and not mirrored, that's true.
You can only afford to lose a vdev, if your vdev itself is mirrored.
You could, for example, create 2 vdev's of raidz2, and instead of striping
them together, mirror them.  Then you could lose one of the raidz2's, and
the other half of the mirror is still up.

Can you layer vdevs hierarchically? Last time I looked, you couldn't
do that.

Regardless, if you want to survive vdev loss, you probably want to set
copies=2 on your data (or 3 if you're exceptionally paranoid), which
could be thought of as another layer of mirroring.

(And you can do this just on the datasets you really want to keep safe,
you don't have to do it on the whole pool.)

[Darren J Moffat]

Post by Peter Tribble

Post by Edward Ned Harvey

Post by Freddie Cash
If you lose 1 vdev, you lose the pool.

As long as 1 vdev is striped and not mirrored, that's true.
You can only afford to lose a vdev, if your vdev itself is mirrored.
You could, for example, create 2 vdev's of raidz2, and instead of striping
them together, mirror them. Then you could lose one of the raidz2's, and
the other half of the mirror is still up.

Can you layer vdevs hierarchically? Last time I looked, you couldn't
do that.

You mean like a mirror of raidz or a raidz or mirrors ?

No the admin interface doesn't allow you to do that.

[Freddie Cash]

Post by Edward Ned Harvey

Post by Freddie Cash
If you lose 1 vdev, you lose the pool.

As long as 1 vdev is striped and not mirrored, that's true.
You can only afford to lose a vdev, if your vdev itself is mirrored.
You could, for example, create 2 vdev's of raidz2, and instead of striping
them together, mirror them.  Then you could lose one of the raidz2's, and
the other half of the mirror is still up.

How does one do that? Is it as simple as using "attach" instead of
"add" when creating the new vdev?

For example:

zpool create mypool raidz2 disk0 disk1 disk2 disk3 disk4 disk5 disk6
zpool attach mypool raidz2 disk7 disk8 disk9 disk10 disk11 disk12 disk13

Would it work for adding a third raidz2 vdev?

[Darren J Moffat]

Post by Freddie Cash

Post by Edward Ned Harvey

Post by Freddie Cash
If you lose 1 vdev, you lose the pool.

As long as 1 vdev is striped and not mirrored, that's true.
You can only afford to lose a vdev, if your vdev itself is mirrored.
You could, for example, create 2 vdev's of raidz2, and instead of striping
them together, mirror them. Then you could lose one of the raidz2's, and
the other half of the mirror is still up.

How does one do that? Is it as simple as using "attach" instead of
"add" when creating the new vdev?
zpool create mypool raidz2 disk0 disk1 disk2 disk3 disk4 disk5 disk6
zpool attach mypool raidz2 disk7 disk8 disk9 disk10 disk11 disk12 disk13
Would it work for adding a third raidz2 vdev?

There is no support in ZFS for nested vdevs like this.

[Freddie Cash]

On Mon, Oct 18, 2010 at 8:51 AM, Darren J Moffat

Post by Darren J Moffat

Post by Edward Ned Harvey

Post by Freddie Cash
If you lose 1 vdev, you lose the pool.

As long as 1 vdev is striped and not mirrored, that's true.
You can only afford to lose a vdev, if your vdev itself is mirrored.
You could, for example, create 2 vdev's of raidz2, and instead of striping
them together, mirror them.  Then you could lose one of the raidz2's, and
the other half of the mirror is still up.

How does one do that?  Is it as simple as using "attach" instead of
"add" when creating the new vdev?
zpool create mypool raidz2 disk0 disk1 disk2 disk3  disk4  disk5  disk6
zpool attach mypool raidz2 disk7 disk8 disk9 disk10 disk11 disk12 disk13
Would it work for adding a third raidz2 vdev?

There is no support in ZFS for nested vdevs like this.

That's what I though, but wanted to make sure. Thanks. :)

[Simon Breden]

So are we all agreed then, that a vdev failure will cause pool loss ?

[Edward Ned Harvey]

Post by Simon Breden
So are we all agreed then, that a vdev failure will cause pool loss ?

Yes. When I said you could mirror a raidzN vdev, it was based on nothing
more credible than assumption backed by suggestion I read somewhere in this
list. Now, the most credible answer comes from all the people telling me
"you're wrong, you can't mirror a raidzN" in this thread. ;-)

[Tuomas Leikola]

Post by Simon Breden
So are we all agreed then, that a vdev failure will cause pool loss ?
--

unless you use copies=2 or 3, in which case your data is still safe
for those datasets that have this option set.

[taemun]

Tuomas:

My understanding is that the "copies" functionality doesn't guarantee that
the extra copies will be kept on a different vdev. So that isn't entirely
true. Unfortunately.

Post by Tuomas Leikola

Post by Simon Breden
So are we all agreed then, that a vdev failure will cause pool loss ?
--

unless you use copies=2 or 3, in which case your data is still safe
for those datasets that have this option set.
--
- Tuomas
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[Cindy Swearingen]

Post by Tuomas Leikola

Post by Simon Breden
So are we all agreed then, that a vdev failure will cause pool loss ?
--

unless you use copies=2 or 3, in which case your data is still safe
for those datasets that have this option set.

This advice is a little too optimistic. Increasing the copies property
value on datasets might help in some failure scenarios, but probably not
in more catastrophic failures, such as multiple device or hardware
failures.

cs

[Bob Friesenhahn]

Post by Cindy Swearingen

Post by Tuomas Leikola
unless you use copies=2 or 3, in which case your data is still safe
for those datasets that have this option set.

This advice is a little too optimistic. Increasing the copies property
value on datasets might help in some failure scenarios, but probably not
in more catastrophic failures, such as multiple device or hardware
failures.

It is 100% too optimistic. The copies option only duplicates the user
data. While zfs already duplicates the metadata (regardless of copies
setting), it is not designed to function if a vdev fails.

Bob

[Tuomas Leikola]

On Wed, Oct 20, 2010 at 3:50 AM, Bob Friesenhahn

Post by Cindy Swearingen

Post by Tuomas Leikola
unless you use copies=2 or 3, in which case your data is still safe
for those datasets that have this option set.

This advice is a little too optimistic. Increasing the copies property
value on datasets might help in some failure scenarios, but probably not
in more catastrophic failures, such as multiple device or hardware
failures.

It is 100% too optimistic.  The copies option only duplicates the user data.
 While zfs already duplicates the metadata (regardless of copies setting),
it is not designed to function if a vdev fails.

Sorry about that. I thought it was already supported with some --force
option. My bad.

[Haudy Kazemi]

Post by Bob Friesenhahn

Post by Cindy Swearingen

Post by Tuomas Leikola
unless you use copies=2 or 3, in which case your data is still safe
for those datasets that have this option set.

This advice is a little too optimistic. Increasing the copies property
value on datasets might help in some failure scenarios, but probably not
in more catastrophic failures, such as multiple device or hardware
failures.

It is 100% too optimistic. The copies option only duplicates the user
data. While zfs already duplicates the metadata (regardless of copies
setting), it is not designed to function if a vdev fails.
Bob

Some future filesystem (not zfs as currently implemented) could be
designed to handle certain vdev failures where multiple vdevs were used
without redundancy at the vdev level. In this scenario, the redundant
metadata and user data with copies=2+ would still be accessible by
virtue of it having been spread across the vdevs, with at least one copy
surviving. Expanding upon this design would allow raw space to be
added, with redundancy being set by a 'copies' parameter.

I understand the copies parameter to currently be designed and intended
as an extra assurance against failures that affect single blocks but not
whole devices. I.e. run ZFS on a laptop with a single hard drive, and
use copies=2 to protect against bad sectors but not complete drive
failures. I have not tested this, however I imagine that performance is
the reason to use copies=2 instead of partitioning/slicing the drive
into two halves and mirroring the two halves back together. I also
recall seeing something about the copies parameter attempting to spread
the copies across different devices, as much as possible.

[Ross Walker]

Post by Tuomas Leikola

Post by Simon Breden
So are we all agreed then, that a vdev failure will cause pool loss ?
--

unless you use copies=2 or 3, in which case your data is still safe
for those datasets that have this option set.

This doesn't prevent pool loss in the face of a vdev failure, merely reduces the likelihood of file loss due to block corruption.

A loss of a vdev (mirror, raidz or non-redundant disk) means the loss of the pool.

-Ross

[Maurice Volaski]

A workaround is to create two pools each with your 1 vdev, make zvols for each of them and export them via iscsi through the localhost interface. Then make a third mirrored pool out of those two iscsi'ed zvols.