OpenSolaris

Printable Version Enter a New Search
Bug ID 6575965
Synopsis panic/thread=2a1016b5ca0: BAD TRAP: type=9 rp=1858500 addr=0 mmu_fsr=0, really, truly out of space
State 10-Fix Delivered (Fix available in build)
Category:Subcategory kernel:zfs
Keywords
Responsible Engineer Neil Perrin
Reported Against
Duplicate Of
Introduced In solaris_nevada
Commit to Fix snv_83
Fixed In snv_83
Release Fixed solaris_nevada(snv_83) , solaris_10u6(s10u6_01) (Bug ID:2160988)
Related Bugs 6608562
Submit Date 2-July-2007
Last Update Date 29-April-2008
Description
bfu to onnv-gate:2007-06-28, while run test suites, zpool_create_011_neg, it hit this panic,

copy the crashdump to 
/net/zion.eng/export/dumps/robin/<bud id>/*.1

panic[cpu13]/thread=2a1016b5ca0: 
BAD TRAP: type=9 rp=1858500 addr=0 mmu_fsr=0

> 2a1016b5ca0::threadlist -v
            ADDR             PROC              LWP CLS PRI            WCHAN
000002a1016b5ca0          184aa40                0   0  60                0
  PC: panicsys+0x48    THREAD: txg_sync_thread()
  stack pointer for thread 2a1016b5ca0: 2a1016b4ce1
    zio_dva_allocate+0x21c()
    zio_write_compress+0x3b4()
    dmu_objset_sync+0x23c()
    dsl_pool_sync+0x8c()
    spa_sync+0x1d0()
    txg_sync_thread+0x190()
    thread_start+4()

> ::stackregs
000002a1016b4ce1 vpanic(7b27b560, 1c, 200, 60012e86080, 2, 4)
  %l0-%l3:      60017921918      60012e860e0         70134a78      60013125c00
  %l4-%l7:               42                0                0                0
  zio_dva_allocate+0x21c:  call      -0x7a1c7608   <panic>

000002a1016b4d91 zio_dva_allocate+0x21c(600180fedc0, 60017921918, 0, 0, 200, 7b27b400)
  %l0-%l3:              200             ffff             fc00      70300000001
  %l4-%l7:         7b27a800                2         7b27b400                2
  zio_write_compress+0x3b4:call      +0x1af0       <zio_next_stage>

000002a1016b4e51 zio_write_compress+0x3b4(600180fedc0, 1, 1, 1, 3, 23e000)
  %l0-%l3:        300000000                3                0                3
  %l4-%l7:            10001            10000                1             ffff
  dmu_objset_sync+0x23c:   call      +0x3816c      <zio_nowait>

000002a1016b4f21 dmu_objset_sync+0x23c(60013260e00, 600149857e0, 60010bbc180, 0, 600145f71c0, 600180fedc0)
  %l0-%l3:         7b212004         7b212000      600154e3f80      60012e86080
  %l4-%l7:                4 ffffffffffffffff                0      60013260e00
  dsl_pool_sync+0x8c:      call      -0x1854       <dsl_dataset_sync>

000002a1016b5031 dsl_pool_sync+0x8c(6001004c2c0, 4, 6001391f200, 6001004c368, 60010bbc180, 6001391f230)
  %l0-%l3:                5                4      6001717abc8      6001004c3f8
  %l4-%l7:      600149857e0      60010e11940      2a1016b5850         7b274000
  spa_sync+0x1d0:          call      -0xb9c4       <dsl_pool_sync>

000002a1016b50e1 spa_sync+0x1d0(6001717a800, 4, 0, 7b275f00, 0, 6001004c428)
  %l0-%l3:      60010bbca80      6001717a988      6001673b1c0      6001004c2c0
  %l4-%l7:      30016f24000             1560      6001717a8f0                8
  txg_sync_thread+0x190:   call      -0x57e0       <spa_sync>

000002a1016b51b1 txg_sync_thread+0x190(6001004c2c0, 4, 7b266ddd, 7b275e28, 701345ec, 2a1016b5a70)
  %l0-%l3:      6001004c3e0      6001004c3d2      6001004c3d0      6001004c390
  %l4-%l7:      6001004c398      6001004c3d6      6001004c3d4      6001004c388
  thread_start+4:          jmpl      %i7, %o7

000002a1016b5291 thread_start+4(6001004c2c0, 0, 0, 0, 0, 0)
  %l0-%l3:                0                0                0                0
  %l4-%l7:                0                0                0                0

There're lots of threads waiting during create the pool with slog.
> 00000600119398c0::walk thread 
30020c93960
30261a5f760
300245583a0
3001f939c40
300245586e0
30039bf10c0
> 30020c93960::threadlist -v
            ADDR             PROC              LWP CLS PRI            WCHAN
0000030020c93960      600119398c0      60010cb7a48   1   0      6001004c3d2
  PC: cv_wait+0x5c    CMD: /usr/sbin/zpool create testpool1.105719 log c1t4d0s1
  stack pointer for thread 30020c93960: 2a100e74d31
  [ 000002a100e74d31 cv_wait+0x5c() ]
    txg_wait_synced+0xf8()
    spa_create+0x3a0()
    zfs_ioc_pool_create+0x44()
    zfsdev_ioctl+0x12c()
    fop_ioctl+0x48()
    ioctl+0x164()
    syscall_trap32+0x1e8()

> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     856672              6692   82%
Anon                        21956               171    2%
Exec and libs                 705                 5    0%
Page cache                    488                 3    0%
Free (cachelist)              674                 5    0%
Free (freelist)            159548              1246   15%

Total                     1040043              8125
Physical                  1021198              7978
sorry, it's zpool_create_011_neg, but not zfs_create_011_pos, updated the prior description.
After investigate, this issue could be easily reproduce by has slog device
but no any toplevel vdevbe specified. such as,

# zpool create tank log c1t4d0s1 (... panic)

As designed, it should return an error of
invalid vdev specification: at least one toplevel vdev must be specified
just like hotspare did,

# zpool create tank spare c1t4d0s1
invalid vdev specification: at least one toplevel vdev must be specified
# echo $?
1

but if change 'spare' to 'log', cause this panic, and the slick metadata
has been updated, it could be observed by 'zdb -l'

# zdb -l /dev/dsk/c1t4d0s1
--------------------------------------------
LABEL 0
--------------------------------------------
    version=8
    name='testpool1.100815'
    state=0
    txg=0
    pool_guid=11861289196565407433
    hostid=2162751173
    hostname='treshnish'
    top_guid=9701111059743535208
    guid=9701111059743535208
    is_log=1
    vdev_tree
        type='disk'
        id=0
        guid=9701111059743535208
        path='/dev/dsk/c1t4d0s1'
        devid='id1,sd@SFUJITSU_MAJ3182M_SUN18G_00P15316____/b'
        phys_path='/sbus@7,0/SUNW,fas@3,8800000/sd@4,0:b'
        whole_disk=0
        metaslab_array=0
        metaslab_shift=0
        ashift=9
        asize=204996608
        is_log=1
    create_txg=4
After recent changes this no longer panics but generates:

Dec  5 14:04:28 mull zfs: WARNING: ZFS: I/O failure (write on <unknown> off 0: zio 30013871310 <hole>): error 28
Dec  5 14:04:28 mull zfs: WARNING: Pool 'whirl' has encountered an uncorrectable I/O error. Manual intervention is required.
Work Around
N/A
Comments
N/A