How can I watch IO operations with dtrace on zfs?

[Trond Norbye]

I have been using iosoop script (see http://www.opensolaris.org/os/community/dtrace/scripts/) written by Brendan Gregg to look at the IO operations of my application. When I was running my test-program on a UFS filesystem I could see both read and write operations like:

UID PID D BLOCK SIZE COMM PATHNAME
203803 4436 R 6016592 16384 diskio <none>
203803 4436 W 3448432 16384 diskio <none>
203803 4436 R 5510832 16384 diskio <none>
203803 4436 W 4652608 16384 diskio <none>

But when I'm running my program on a ZFS file-system, I can only see the read io transfers:

UID PID D BLOCK SIZE COMM PATHNAME
203803 4445 R 2647296 131072 diskio <none>
203803 4445 R 1351168 131072 diskio <none>

So how can I get the same information from a ZFS file-system?

The reason I'm asking is because I'm trying to see how IO operations behave on the different filesystems.

Any help / pointers would be appreciated.

Trond


This message posted from opensolaris.org

[Brendan Gregg]

G'Day Trond,

Post by Trond Norbye
I have been using iosoop script (see
http://www.opensolaris.org/os/community/dtrace/scripts/) written by
Brendan Gregg to look at the IO operations of my application. When I was
running my test-program on a UFS filesystem I could see both read and

The most up to date version would be in the DTraceToolkit, but that
shouldn't cause a problem here.

Post by Trond Norbye
UID PID D BLOCK SIZE COMM PATHNAME
203803 4436 R 6016592 16384 diskio <none>
203803 4436 W 3448432 16384 diskio <none>
203803 4436 R 5510832 16384 diskio <none>
203803 4436 W 4652608 16384 diskio <none>
UID PID D BLOCK SIZE COMM PATHNAME
203803 4445 R 2647296 131072 diskio <none>
203803 4445 R 1351168 131072 diskio <none>
So how can I get the same information from a ZFS file-system?

Firstly, we need to bear in mind that iosnoop matches on disk I/O, not
application level I/O. Many writes will be cached in memory and flushed
sometime later to disk, asynchronously. That's when iosnoop picks it up.
Were you matching on a process name or PID?

no worries,

Brendan

[Matthew Ahrens]

Post by Trond Norbye
I have been using iosoop script (see
http://www.opensolaris.org/os/community/dtrace/scripts/) written by
Brendan Gregg to look at the IO operations of my application.

...

Post by Trond Norbye
So how can I get the same information from a ZFS file-system?

As you can see, ZFS is not yet fully integrated with the dtrace i/o
provider. With ZFS, writes are (typically) deferred, so it is
nontrivial to assign each write i/o to a particular application. If you
are familiar with dtrace, you can use fbt to look at the zio_done()
function, eg. with something like this:

zio_done:entry
/args[0]->io_type == 1 && args[0]->io_bp != NULL/
{
@bytes["read",
args[0]->io_bookmark.zb_objset,
args[0]->io_bookmark.zb_object,
args[0]->io_bookmark.zb_level,
args[0]->io_bookmark.zb_blkid != 0] =
/* sum(args[0]->io_size); */
count();
}

zio_done:entry
/args[0]->io_type == 2/
{
@bytes["write",
args[0]->io_bookmark.zb_objset,
args[0]->io_bookmark.zb_object,
args[0]->io_bookmark.zb_level,
args[0]->io_bookmark.zb_blkid != 0] =
/* sum(args[0]->io_size); */
count();
}

END
{
printf("r/w objset object level blk>0 i/os\n");
printa("%5s %4d %7d %d %d %@d\n", @bytes);
printf("r/w objset object level blk>0 i/os\n");
}

--matt