Discussion:
how to know available disk space
(too old to reply)
Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
2013-02-06 23:26:44 UTC
Permalink
I have a bunch of VM's, and some samba shares, etc, on a pool. I created the VM's using zvol's, specifically so they would have an appropriate refreservation and never run out of disk space, even with snapshots. Today, I ran out of disk space, and all the VM's died. So obviously it didn't work.

When I used "zpool status" after the system crashed, I saw this:
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 568G 360G - 61% 1.00x ONLINE -

I did some cleanup, so I could turn things back on ... Freed up about 4G.

Now, when I use "zpool status" I see this:
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 564G 364G - 60% 1.00x ONLINE -

When I use "zfs list storage" I see this:
NAME USED AVAIL REFER MOUNTPOINT
storage 909G 4.01G 32.5K /storage

So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in "usable," as in, "how much can I write before I run out of space," you should use "zfs list" and not "zpool status"
Gregg Wonderly
2013-02-07 01:17:14 UTC
Permalink
This is one of the greatest annoyances of ZFS. I don't really understand how, a zvol's space can not be accurately enumerated from top to bottom of the tree in 'df' output etc. Why does a "zvol" divorce the space used from the root of the volume?

Gregg Wonderly
I have a bunch of VM's, and some samba shares, etc, on a pool. I created the VM's using zvol's, specifically so they would have an appropriaterefreservation and never run out of disk space, even with snapshots. Today, I ran out of disk space, and all the VM's died. So obviously it didn't work.
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 568G 360G - 61% 1.00x ONLINE -
I did some cleanup, so I could turn things back on ... Freed up about 4G.
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 564G 364G - 60% 1.00x ONLINE -
NAME USED AVAIL REFER MOUNTPOINT
storage 909G 4.01G 32.5K /storage
So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in "usable," as in, "how much can I write before I run out of space," you should use "zfs list" and not "zpoolstatus"
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
2013-02-07 02:10:11 UTC
Permalink
This is one of the greatest annoyances of ZFS.  I don't really understand how,
a zvol's space can not be accurately enumerated from top to bottom of the
tree in 'df' output etc.  Why does a "zvol" divorce the space used from the
root of the volume?
The way I would say that is: Intuitively, I think people expect reservations to count against Alloc and Used.
Richard Elling
2013-02-07 17:39:32 UTC
Permalink
Post by Gregg Wonderly
This is one of the greatest annoyances of ZFS. I don't really understand how, a zvol's space can not be accurately enumerated from top to bottom of the tree in 'df' output etc. Why does a "zvol" divorce the space used from the root of the volume?
Thick (with reservation) or thin provisioning behave differently. Also, depending on
how you created the reservation, it might or might not account for the metadata
overhead needed. By default, space for metadata is reserved, but if you use the
-s (sparse aka thin provisioning) option, then later reservation changes are set as
absolute.

Also, metadata space, compression, copies, and deduplication must be accounted for.
The old notions of free/available don't match very well with these modern features.
Post by Gregg Wonderly
Gregg Wonderly
I have a bunch of VM's, and some samba shares, etc, on a pool. I created the VM's using zvol's, specifically so they would have an appropriaterefreservation and never run out of disk space, even with snapshots. Today, I ran out of disk space, and all the VM's died. So obviously it didn't work.
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 568G 360G - 61% 1.00x ONLINE -
I did some cleanup, so I could turn things back on ... Freed up about 4G.
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 564G 364G - 60% 1.00x ONLINE -
NAME USED AVAIL REFER MOUNTPOINT
storage 909G 4.01G 32.5K /storage
So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in "usable," as in, "how much can I write before I run out of space," you should use "zfs list" and not "zpoolstatus"
Correct. zpool status does not show dataset space available.
-- richard

--

***@RichardElling.com
+1-760-896-4422
Jan Owoc
2013-02-07 03:03:13 UTC
Permalink
On Wed, Feb 6, 2013 at 4:26 PM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris)
Post by Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 568G 360G - 61% 1.00x ONLINE -
I did some cleanup, so I could turn things back on ... Freed up about 4G.
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 564G 364G - 60% 1.00x ONLINE -
NAME USED AVAIL REFER MOUNTPOINT
storage 909G 4.01G 32.5K /storage
So I guess the lesson is (a) refreservation and zvol alone aren't enough to
ensure your VM's will stay up. and (b) if you want to know how much room is
*actually* available, as in "usable," as in, "how much can I write before I
run out of space," you should use "zfs list" and not "zpool status"
Could you run "zfs list -o space storage"? It will show how much is
used by the data, the snapshots, refreservation, and children (if
any). I read somewhere that one should always use "zfs list" to
determine how much space is actually available to be written on a
given filesystem.

I have an idea, but it's a long shot. If you created more than one zfs
on that pool, and added a reservation to each one, then that space is
still technically unallocated as far as "zpool list" is concerned, but
is not available to writing when you do "zfs list". I would imagine
you have one or more of your VMs that grew outside of their
"refreservation" and now crashed for lack of free space on their zfs.
Some of the other VMs aren't using their refreservation (yet), so they
could, between them, still write 360GB of stuff to the drive.

Jan
Pasi Kärkkäinen
2013-02-08 15:09:56 UTC
Permalink
Post by Jan Owoc
On Wed, Feb 6, 2013 at 4:26 PM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris)
Post by Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 568G 360G - 61% 1.00x ONLINE -
I did some cleanup, so I could turn things back on ... Freed up about 4G.
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
storage 928G 564G 364G - 60% 1.00x ONLINE -
NAME USED AVAIL REFER MOUNTPOINT
storage 909G 4.01G 32.5K /storage
So I guess the lesson is (a) refreservation and zvol alone aren't enough to
ensure your VM's will stay up. and (b) if you want to know how much room is
*actually* available, as in "usable," as in, "how much can I write before I
run out of space," you should use "zfs list" and not "zpool status"
Could you run "zfs list -o space storage"? It will show how much is
used by the data, the snapshots, refreservation, and children (if
any). I read somewhere that one should always use "zfs list" to
determine how much space is actually available to be written on a
given filesystem.
I have an idea, but it's a long shot. If you created more than one zfs
on that pool, and added a reservation to each one, then that space is
still technically unallocated as far as "zpool list" is concerned, but
is not available to writing when you do "zfs list". I would imagine
you have one or more of your VMs that grew outside of their
"refreservation" and now crashed for lack of free space on their zfs.
Some of the other VMs aren't using their refreservation (yet), so they
could, between them, still write 360GB of stuff to the drive.
I'm seeing weird output aswell:

# zpool list foo
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
foo 5.44T 4.44T 1023G 81% 14.49x ONLINE -

# zfs list | grep foo
foo 62.9T 0 250G /volumes/foo
foo/.nza-reserve 31K 100M 31K none
foo/foo 62.6T 0 62.6T /volumes/foo/foo

# zfs list -o space foo
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
foo 0 62.9T 0 250G 0 62.7T

# zfs list -o space foo/foo
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
foo/foo 0 62.6T 0 62.6T 0 0


What's the correct way of finding out what actually uses/reserves that 1023G of FREE in the zpool?

At this point the filesystems are full, and it's not possible to write to them anymore.
Also creating new filesystems to the pool fail:

"Operation completed with error: cannot create 'foo/Test': out of space"

So the zpool is full for real.

I'd like to better understand what actually uses that 1023G of FREE space reported by zpool..
1023G out of 4.32T is around 22% overhead..
zpool "foo" consists of 3x mirror vdevs, so there's no raidz involved.

62.6T / 14.49x dedup-ratio = 4.32T
Which is pretty close to the ALLOC value reported by zpool..

Data on the filesystem is VM images written over NFS.


Thanks,

-- Pasi
Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
2013-02-08 21:47:38 UTC
Permalink
Post by Pasi Kärkkäinen
What's the correct way of finding out what actually uses/reserves that 1023G
of FREE in the zpool?
Maybe this isn't exactly what you need, but maybe:

for fs in `zfs list -H -o name` ; do echo $fs ; zfs get reservation,refreservation,usedbyrefreservation $fs ; done
Post by Pasi Kärkkäinen
At this point the filesystems are full, and it's not possible to write to them anymore.
You'll have to either reduce your reservations, or destroy old snapshots. Or add more disks.
Pasi Kärkkäinen
2013-02-08 22:02:13 UTC
Permalink
Post by Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
Post by Pasi Kärkkäinen
What's the correct way of finding out what actually uses/reserves that 1023G
of FREE in the zpool?
for fs in `zfs list -H -o name` ; do echo $fs ; zfs get reservation,refreservation,usedbyrefreservation $fs ; done
I checked this and there are no reservations configured (or well, there are the 100MB defaults, but not more than that). So reservations don't explain this..
Post by Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
Post by Pasi Kärkkäinen
At this point the filesystems are full, and it's not possible to write to them anymore.
You'll have to either reduce your reservations, or destroy old snapshots. Or add more disks.
There aren't any snapshots either.. I know adding disks will fix the problem,
but I'd like to understand why zpool says there is almost 1TB of FREE space when clearly there isn't..


Thanks for the reply!

-- Pasi
Jim Klimov
2013-02-08 22:44:21 UTC
Permalink
On 2013-02-08 22:47, Edward Ned Harvey
Post by Edward Ned Harvey (opensolarisisdeadlongliveopensolaris)
for fs in `zfs list -H -o name` ; do echo $fs ; zfs get reservation,refreservation,usedbyrefreservation $fs ; done
What is the sacramental purpose of such construct in comparison to:

zfs list -H -o reservation,refreservation,usedbyrefreservation,name \
-t filesystem {-r pool/interesting/dataset}

Just asking - or suggesting a simpler way to do stuff ;)

//Jim
Pasi Kärkkäinen
2013-02-20 09:31:49 UTC
Permalink
Hello,

Any comments/suggestions about this would be very nice..

Thanks!

-- Pasi
Post by Pasi Kärkkäinen
# zpool list foo
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
foo 5.44T 4.44T 1023G 81% 14.49x ONLINE -
# zfs list | grep foo
foo 62.9T 0 250G /volumes/foo
foo/.nza-reserve 31K 100M 31K none
foo/foo 62.6T 0 62.6T /volumes/foo/foo
# zfs list -o space foo
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
foo 0 62.9T 0 250G 0 62.7T
# zfs list -o space foo/foo
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
foo/foo 0 62.6T 0 62.6T 0 0
What's the correct way of finding out what actually uses/reserves that 1023G of FREE in the zpool?
At this point the filesystems are full, and it's not possible to write to them anymore.
"Operation completed with error: cannot create 'foo/Test': out of space"
So the zpool is full for real.
I'd like to better understand what actually uses that 1023G of FREE space reported by zpool..
1023G out of 4.32T is around 22% overhead..
zpool "foo" consists of 3x mirror vdevs, so there's no raidz involved.
62.6T / 14.49x dedup-ratio = 4.32T
Which is pretty close to the ALLOC value reported by zpool..
Data on the filesystem is VM images written over NFS.
Thanks,
-- Pasi
Loading...