That does help tremendously. How does ZFS decide which zio cache to
use? I apologize if this has already been addressed somewhere.
Post by Mark MaybeePost by Al HopperPost by Mark MaybeePost by Jason J. W. WilliamsHi Robert,
Thank you! Holy mackerel! That's a lot of memory. With that type of a
calculation my 4GB arc_max setting is still in the danger zone on a
Thumper. I wonder if any of the ZFS developers could shed some light
on the calculation?
In a worst-case scenario, Robert's calculations are accurate to a
certain degree: If you have 1GB of dnode_phys data in your arc cache
(that would be about 1,200,000 files referenced), then this will result
in another 3GB of "related" data held in memory: vnodes/znodes/
dnodes/etc. This related data is the in-core data associated with
an accessed file. Its not quite true that this data is not evictable,
it *is* evictable, but the space is returned from these kmem caches
only after the arc has cleared its blocks and triggered the "free" of
the related data structures (and even then, the kernel will need to
to a kmem_reap to reclaim the memory from the caches). The
fragmentation that Robert mentions is an issue because, if we don't
free everything, the kmem_reap may not be able to reclaim all the
memory from these caches, as they are allocated in "slabs".
We are in the process of trying to improve this situation.
.... snip .....
Understood (and many Thanks). In the meantime, is there a rule-of-thumb
that you could share that would allow mere humans (like me) to calculate
the best values of zfs:zfs_arc_max and ncsize, given the that machine has
a) a busy NFS server
b) a general multiuser development server
c) a database server
d) an Apache/Tomcat/FTP server
e) a single user Gnome desktop running U3 with home dirs on a ZFS
filesystem
It would seem, from reading between the lines of previous emails,
particularly the ones you've (Mark M) written, that there is a rule of
thumb that would apply given a standard or modified ncsize tunable??
I'm primarily interested in a calculation that would allow settings that
would reduce the possibility of the machine descending into "swap hell".
Ideally, there would be no need for any tunables; ZFS would always "do
the right thing". This is our grail. In the meantime, I can give some
recommendations, but there is no "rule of thumb" that is going to work
in all circumstances.
ncsize: As I have mentioned previously, there are overheads
associated with caching "vnode data" in ZFS. While
the physical on-disk data for a znode is only 512bytes,
the related in-core cost is significantly higher.
Roughly, you can expect that each ZFS vnode held in
the DNLC will cost about 3K of kernel memory.
So, you need to set ncsize appropriately for how much
memory you are willing to devote to it. 500,000 entries
is going to cost you 1.5GB of memory.
zfs_arc_max: This is the maximum amount of memory you want the
ARC to be able to use. Note that the ARC won't
necessarily use this much memory: if other applications
need memory, the ARC will shrink to accommodate.
Although, also note that the ARC *can't* shrink if all
of its memory is held. For example, data in the DNLC
cannot be evicted from the ARC, so this data must first
be evicted from the DNLC before the ARC can free up
space (this is why it is dangerous to turn off the ARCs
ability to evict vnodes from the DNLC).
Also keep in mind that the ARC size does not account for
many in-core data structures used by ZFS (znodes/dnodes/
dbufs/etc). Roughly, for every 1MB of cached file
pointers, you can expect another 3MB of memory used
outside of the ARC. So, in the example above, where
ncsize is 500,000, the ARC is only seeing about 400MB
of the 1.5GB consumed. As I have stated previously,
we consider this a bug in the current ARC accounting
that we will soon fix. This is only an issue in
environments where many files are being accessed. If
the number of files accessed is relatively low, then
the ARC size will be much closer to the actual memory
consumed by ZFS.
So, in general, you should not really need to "tune"
zfs_arc_max. However, in environments where you have
specific applications that consume known quantities of
memory (e.g. database), it will likely help to set the
ARC max size down, to guarantee that the necessary
kernel memory is available. There may be other times
when it will be beneficial to explicitly set the ARCs
maximum size, but at this time I can't offer any general
"rule of thumb".
I hope that helps.
-Mark
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
