ZFS currently does a poor job of recognizing devices which have
been removed from the system. This is only supported by some
drivers, and doesn't result in a particularly friendly error
message.
We need to distinguish between device removal and arbitrary
failure. After much research into various dark corners of the
I/O subsystem, it seems like the appropriate way to detect this
is to issue the DKIOCSTATE ioctl() in response to I/O errors.
Modulo some bugs in the current sd(7d) implementation, this will
return DKIO_DEV_GONE if there is no longer any physical
connection to the disk.
ZFS will then close the device and put into a new vdev state,
VDEV_STATE_REMOVED. By virtue of closing the device, this will
allow the device node to be torn down and for the rest of the
cfgadm(1M) framework to take over.