raidz and compression, difficulties

[Joachim Pihl]

Running SXDE (snv_70) for a file server, and I must admit I'm new to
Solaris and zfs. zfs does not appear to do any compression at all, here is
what I did to set it up:

I created a four drive raidz array:

zpool create pool raidz c0d0 c0d1 c1d0 c1d1

then an extra file system on that:

zfs create -o mountpoint=/export/data pool/data

After adding a bit of data (200GB or so), i figured that compression would
in fact be a good idea:

zfs set compression=on pool/data

So far so good, "zfs get all" reports compression to be active. Now for
the problem: After adding another 300GB of uncompressed .tif and .bin/.cue
(audio CD) files, compression ratio is still at 1.00, indicating that no
compression has taken place. Where did I go wrong? Anything I can do to
get compression back after the cause for the problem has been fixed? All
the data has been copied across a network, through Samba - set up
according to instructions found in various Solaris forums.

[Joachim Pihl]

it won't go back and compress old data, but if you copy
the files, perhaps into a new folder, they will be
compressed on that copy or move.

I can't see why that should help, since the 300GB I added after enabling
compression hasn't been compressed at all. The compression is only
seemingly enabled, is this a bug with build 70, or is there some
misconfiguration at my place?

[Toby Thain]

Post by Joachim Pihl
Running SXDE (snv_70) for a file server, and I must admit I'm new to
Solaris and zfs. zfs does not appear to do any compression at all, here is
zpool create pool raidz c0d0 c0d1 c1d0 c1d1
zfs create -o mountpoint=/export/data pool/data
After adding a bit of data (200GB or so), i figured that
compression would
zfs set compression=on pool/data
So far so good, "zfs get all" reports compression to be active. Now for
the problem: After adding another 300GB of uncompressed .tif
and .bin/.cue
(audio CD) files,

I wouldn't expect very much compression on that kind of data.

--Toby

Post by Joachim Pihl
compression ratio is still at 1.00, indicating that no
compression has taken place. Where did I go wrong? Anything I can do to
get compression back after the cause for the problem has been
fixed? All
the data has been copied across a network, through Samba - set up
according to instructions found in various Solaris forums.
--
Joachim
Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter?
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[Joachim Pihl]

Post by Toby Thain

Post by Joachim Pihl
So far so good, "zfs get all" reports compression to be active. Now for
the problem: After adding another 300GB of uncompressed .tif and .bin/.cue
(audio CD) files,

I wouldn't expect very much compression on that kind of data.

I didn't expect miracles, but since WinRAR gave 13% compression on a
representative data set with ZIP compression at the fastest setting, I was
expecting to see a compression ratio > 1.05 at least, not == 1.00. Getting
5-10% more space on a drive never hurts.

Could it be that I should specify compression algorithm? Documentation
says it will use lzjb by default, so I did not set it explicitly.

[MC]

Post by Joachim Pihl
I didn't expect miracles, but since WinRAR gave 13% compression

ZFS doesn't compress a block if it can't get a certain amount of return on it.

Since the default compression is less effective than RAR, you can bet ZFS is seeing much less than 13% return.

I expect everything is working properly, and you are just dealing with well-compressed files :)

You could try setting the zfs compression to a stronger algorithm to see if anything gets compressed that way.


This message posted from opensolaris.org

[Joachim Pihl]

Post by MC

Post by Joachim Pihl
I didn't expect miracles, but since WinRAR gave 13% compression

ZFS doesn't compress a block if it can't get a certain amount of return on it.
Since the default compression is less effective than RAR, you can bet
ZFS is seeing much less than 13% return.

What is the threshold, does anyone know that? It sound like a fair
decision, though. Try to compress on write, increase perrformance on read
by not compressing if there is little space to be gained. Wouldn't slow
down writes compared to not having such a threshold either. Clever!

Post by MC
I expect everything is working properly, and you are just dealing with
well-compressed files :)

Raw audio and uncompressed 16bit TIFF files (35mm negative scans) have
high content entropy, so they are difficult to compress, that could be the
culprit.

Post by MC
You could try setting the zfs compression to a stronger algorithm to see
if anything gets compressed that way.

[Darren J Moffat]

Post by Joachim Pihl

Post by MC

Post by Joachim Pihl
I didn't expect miracles, but since WinRAR gave 13% compression

ZFS doesn't compress a block if it can't get a certain amount of return on it.
Since the default compression is less effective than RAR, you can bet
ZFS is seeing much less than 13% return.

What is the threshold, does anyone know that? It sound like a fair
decision, though. Try to compress on write, increase perrformance on read
by not compressing if there is little space to be gained. Wouldn't slow
down writes compared to not having such a threshold either. Clever!

See line 108 of this file:

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/zio_compress.c#73

[Tim]

Why not create a sample dataset and throw some large .txt files out there
and see what happens? That way you'll know for certain if there's some bug
you're hitting, or if it's just not applicable to your current dataset.

Post by Joachim Pihl

Post by Toby Thain

Post by Joachim Pihl
So far so good, "zfs get all" reports compression to be active. Now for
the problem: After adding another 300GB of uncompressed .tif and .bin/.cue
(audio CD) files,

I wouldn't expect very much compression on that kind of data.

I didn't expect miracles, but since WinRAR gave 13% compression on a
representative data set with ZIP compression at the fastest setting, I was
expecting to see a compression ratio > 1.05 at least, not == 1.00. Getting
5-10% more space on a drive never hurts.
Could it be that I should specify compression algorithm? Documentation
says it will use lzjb by default, so I did not set it explicitly.
--
Joachim
Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter?
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[Joachim Pihl]

Post by Tim
Why not create a sample dataset and throw some large .txt files out there
and see what happens? That way you'll know for certain if there's some bug
you're hitting, or if it's just not applicable to your current dataset.

Good idea, but not tonight... It try two things: creating a few dozen GBs
worth of "The quick brown fox...", and then create a temporary file
system, enable the compression _before_ adding anything to it, then copy
the same data to the new file system to see if I can spot any difference.

Is there any way to check the compression of a single file in zfs, btw?
Would come in handy for this task, and sifting through tons of man files
is _not_ my favorite hobby. :-)

[Dale Ghent]

Post by Joachim Pihl
So far so good, "zfs get all" reports compression to be active. Now for
the problem: After adding another 300GB of uncompressed .tif
and .bin/.cue
(audio CD) files, compression ratio is still at 1.00, indicating that no
compression has taken place.

TIFF files can how their own compression ("compressed TIFF") and many
image editors have this on by default, so you wouldn't know about it
unless you specifically looked for it. I think the TIFF spec specifies
LZW compression for this... but either way, if this is indeed the
case, zfs compression won't help with those

Now, the bin/cue file format specifies no optional compression, so
those bin files should be nothing but a raw image of 16bit PCM
audio... which you should see some (but not great) compression with.
The default lzjb compression scheme in zfs might not be terribly
effective on this type of file data being that it's optimized for
speed rather than compression efficiency. Try turning on gzip
compression in zfs instead and see if things improve.

To make that simple I'd just make a new fs (eg; pool/data/audio) and
then 'zfs set compression=gzip-4 pool/data/audio' and then mv your bin/
cue files there.

/dale