OpenSolaris

Category
   kernel
Sub-Category
   zfs
Description
   raidz will not replace a vdev that was previously pulled during replacement, instead leaving 2 devices under replacing, and the pool in a degraded state (disk was zeroed) after removal: 
zpool status
  pool: data
 state: DEGRADED
 scrub: none requested
config:
        NAME                        STATE     READ WRITE CKSUM
        data                        DEGRADED     0     0     0
          raidz1                    DEGRADED     0     0     0
            c0t0d0                  ONLINE       0     0     0
            c0t1d0                  ONLINE       0     0     0
            replacing               UNAVAIL      0   437     0  insufficient replicas
              17096229131581286394  UNAVAIL      0   497     0  was /dev/dsk/c0t2d0s0/old
              11342560969745958696  UNAVAIL      0   498     0  was /dev/dsk/c0t2d0s0
            c0t3d0                  ONLINE       0     0     0
            c0t4d0                  ONLINE       0     0     0
            c0t5d0                  ONLINE       0     0     0
            c0t6d0                  ONLINE       0     0     0
            c0t7d0                  ONLINE       0     0     0
            c2t2d0                  ONLINE       0     0     0
            c2t3d0                  ONLINE       0     0     0
errors: No known data errors
after moving disk to (c2t1d0)
#zpool replace data 17096229131581286394 c2t1d0
cannot replace 17096229131581286394 with c2t1d0: cannot replace a replacing device
# zpool replace data 11342560969745958696 c2t1d0
cannot replace 11342560969745958696 with c2t1d0: cannot replace a replacing device
also occurs when attempting to offline:
# zpool offline data 17096229131581286394
cannot offline 17096229131581286394: no valid replicas
# zpool offline data 11342560969745958696
cannot offline 11342560969745958696: no valid replicas
Frequency
   Always
Regression
   No
Steps to Reproduce
   create a raidz
remove disk and zero
reattach disk 
begin zpool replace operation
cancel zpool scrub during replacement
rezero disk or otherwise alter pool config info on it
problem occurs here when attempting to use zpool replace or zpool online on it
Expected Result
   zpool replace should allow replacement of the vdev once it realizes that either one of the vdevs marked "replacing" are not actually present or are faulted/oflline. 
Actual Result
   #zpool replace data 17096229131581286394 c2t1d0
cannot replace 17096229131581286394 with c2t1d0: cannot replace a replacing device
# zpool replace data 11342560969745958696 c2t1d0
cannot replace 11342560969745958696 with c2t1d0: cannot replace a replacing device
Error Message(s)
   cannot replace [vdevname] with [newvdevname]: cannot replace a replacing device
Test Case
   
Workaround
   
Submitter wants to work on bug
   No
Additional configuration information
   10 disk raidz1, 8 drives on an mpt based controller (lsi1068), 2 on an ich7
i can provide additional info if needed

Detach one of the two failed disks, and do a replace on the other.

e.g.:

# zpool detach data 17096229131581286394
# zpool replace data 11342560969745958696 c2t1d0

N/A