Error 16: Inconsistent filesystem structure after a change in the system

[Rafal Pratnicki]

I've hit this bug on my home machine couple of times and finally decided to log
it since I've spend 2 days configuring my "OpenSolaris 2008.11 snv_101b_rc2
X86" and after installing the SUNWsmbfskr package I ended up in the grub> menu.
The package contains a necessary module for CIFS. After pkg install
SUNWsmbfskr, I rebooted the box. There was no grub menu anymore but only the
grub prompt. I managed to boot it manually using another BE. I was unable to
use the original BE getting this error message during the manual boot:
bootfs rpool/ROOT/rp01
OK
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
OK
module$ /platform/i86pc/$ISADIR/boot_archive
after this command I got the error: Error 16: Inconsistent filesystem
structure.
I had to use a different BE to boot up the system.
I've tried to repair boot_archive from the rp01 BE but unsuccessfully every
time ending up in the grub menu forced to manually boot the system using a
different BE.
My unsuccessful method to rebuild boot_archive:
# mkdir /a
# bootadm mount rp01 /a
# rm /a/platform/i86pc/boot_archive
# bootadm update-archive -f -R /a
Creating boot_archive for /a
updating /a/platform/i86pc/amd64/boot_archive
updating /a/platform/i86pc/boot_archive
# beadm unmount rp01
# reboot

***@panda:~$ beadm list
BE Active Mountpoint Space Policy Created
-- ------ ---------- ----- ------ -------
opensolaris - - 8.23M static 2008-12-16 21:04
opensolaris-1 - - 121.78M static 2008-12-20 12:40
opensolaris-2 - - 8.21M static 2008-12-21 01:57
opensolaris-3 - - 67.31M static 2008-12-21 15:12
opensolaris-4 - - 306.0K static 2008-12-21 20:46
opensolaris-5 - - 165.07M static 2008-12-21 20:49
rp01 - - 140.59M static 2008-12-22 21:38
rp02 NR / 3.48G static 2008-12-23 00:06
update - - 13.59M static 2008-12-20 12:25

To reproduce the bug on my system is very simple just change/install/un-install
anything in the kernel directory or try to create, activate a new BE using the
beadm command and you will end up in the grub prompt (no grub menu available).
To rescue from this stage only a manual boot or an Osol2008.11 Live CD.

Could you please provide me with a method to recover the rp01 BE. I've spend to
much time configuring it and don't want to loose the changes. But the method
must be different then the one I provided above as the above one doesn't work.

Also trying to create a new BE from the rp01 BE doesn't work:
***@panda:~$ pfexec su -
Sun Microsystems Inc. SunOS 5.11 snv_101b November 2008
You have new mail.
***@panda:~# export BE_PRINT_ERR=true
***@panda:~# beadm create -e rp01 rp03
be_copy: failed to snapshot BE (rpool/ROOT/***@rp03): dataset is busy
Unable to create rp03.
Mount busy.
***@panda:~#

As you can see the rp01 BE suppose to be a non-active BE name but somehow it's
not.

I've checked the disk and there are no errors:
***@panda:~$ zpool status -xv
all pools are healthy
***@panda:~$ zpool status
pool: rpool
state: ONLINE
scrub: scrub completed after 2h51m with 0 errors on Mon Dec 29 02:58:19 2008
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c3d0s0 ONLINE 0 0 0

errors: No known data errors
***@panda:~$

The next day I recovered the system meaning I have the grub menu and the system boots up
fine automatically without any manual intervention. After that the update
manager recognized that on pkg.opensolaris.org there is an update for my
system. It was the entire package v0.5.11-0.101. So I clicked the update all
button, a new window popped up "Create new boot environment". This should
create an update clone of the active BE. After a reboot the same story. No grub
menu available, only the grub> prompt. The system didn't boot automatically
anymore.
The error message is the same: Error 16: Inconsistent filesystem structure

[Rafal Pratnicki]

I recovered the system and created the opensolaris-12 BE. The system was working fine. I had the grub menu, it was fully recovered.
At this stage I decided to create a new BE but leave the opensolaris-12 BE as an active BE and manually boot to the opensolaris-13 BE.
So the situation looked like this:
beadm list
opensolaris-12 R / 3.73G static 2009-01-03 14:44
opensolaris-13 N - 120.74M static 2009-01-03 14:51
Running the opensolaris-13 BE I installed 2 packages: SUNWsmbs SUNWsmbskr responsible for the solaris CIFS service. I reboot the box and it still was fine. I had the grub menu and the boot_archive wasn't corrupted. After the reboot I configured the solaris CIFS service and exchanged some data between the opensolaris and a laptop via a network. I've been doing this for 3h. Then I decided to reboot the box again to see if it's still OK. I used init 6, but this time I ended up in the grub> menu. When I manually wanted to boot it looked like this:
bootfs rpool/ROOT/opensolaris-13
OK
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
Error 16: Inconsistent filesystem structure
I had to use rpool/ROOT/opensolaris-12 and it was fine.
I didn't even get to load the boot_archive module. Usually it breaks during the boot_archive module load.
I'm unable to use the opensolaris-13 BE and I've lost the CIFS configuration as I can't boot the BE.

Can someone wise help me to narrow down this bug.
Thanks in advance
Rafal

[Jan Spitalnik]

Hey Rafal,

this sounds like missing GANG block support in GRUB. Checkout putback
log for snv_106 (afaik), there's a bug where grub fails like this.

Cheers,
Spity

Post by Rafal Pratnicki
I recovered the system and created the opensolaris-12 BE. The system
was working fine. I had the grub menu, it was fully recovered.
At this stage I decided to create a new BE but leave the
opensolaris-12 BE as an active BE and manually boot to the
opensolaris-13 BE.
beadm list
opensolaris-12 R / 3.73G static 2009-01-03 14:44
opensolaris-13 N - 120.74M static 2009-01-03 14:51
Running the opensolaris-13 BE I installed 2 packages: SUNWsmbs
SUNWsmbskr responsible for the solaris CIFS service. I reboot the
box and it still was fine. I had the grub menu and the boot_archive
wasn't corrupted. After the reboot I configured the solaris CIFS
service and exchanged some data between the opensolaris and a laptop
via a network. I've been doing this for 3h. Then I decided to reboot
the box again to see if it's still OK. I used init 6, but this time
I ended up in the grub> menu. When I manually wanted to boot it
bootfs rpool/ROOT/opensolaris-13
OK
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
Error 16: Inconsistent filesystem structure
I had to use rpool/ROOT/opensolaris-12 and it was fine.
I didn't even get to load the boot_archive module. Usually it breaks
during the boot_archive module load.
I'm unable to use the opensolaris-13 BE and I've lost the CIFS
configuration as I can't boot the BE.
Can someone wise help me to narrow down this bug.
Thanks in advance
Rafal
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss