OpenSolaris

opensolaris.org has a custom PAM module that sets up a chroot environment for access to the source repositories that are hosted there.  As described at http://blogs.sun.com/kupfer/entry/unwanted_mounts, we've recently discovered that the session-close routine no longer runs as root, which breaks our PAM module.  The blog entry has this comment from Nico:

	Actually, when I did the SunSSH resync back in the S10 days I
	made sure that pam_close_session() and pam_end() were called
	as root, with all privs. That was part of the SunSSH
	altprivsep model. However, that was recently broken as a
	result of a change to better use the Solaris crypto framework,
	and fixing it again will be very hard this time because of
	fork-safety issues in PKCS#11 and other things.

N/A

yes, Nico caught that during the code review and after some discussion it was deemed OK to keep it that way for the time being. As said, it will be very hard to fix and I'm not sure even how at this point.

we would need the Crypto Framework to implement C_GetOperationState for symetric crypto as well, at least. It is implemented for MD5 and SHA-1 only now. Or, we would need to be able to save the PAM state and export it to another process (monitor). I'm not aware that this would be possible either.

I'm not sure we can fix that in SunSSH without totally rewriting the privilege separation code, or to do a key re-exchange for every connection right after the authentication. The latter one is probably no go, it would seriously affect the authentication time. The former seems like a very big thing now.
after a further discussion it seems that we might make use of a partially grey area in how PKCS#11 sessions should be worked with after the fork. The spec suggests that any attempt to use sessions in the child without reinitializing the engine after fork should return cryptoki-uninitilized like error, but states that the actual behaviour is unspecified.

so, we could add a flag to the engine that if used it would tell the engine that parent will not use any sessions after the fork, and that the child will take over. That way, we could do the authentication in the monitor and fork the unprivileged child after that, as it was before the change for SunSSH HW crypto support. That way we would be backward compatible and the changed behaviour would be trigger manually from the application (parent).

for setting the flag we could use the ENGINE control API. Obviously, the application would have to honour such behaviour and the parent would not be allowed to touch any existing sessions or engine references after that.
What was suggested in the previous entry can not be done. libpkcs11 itself has atfork() handlers so any existing PKCS#11 sessions are invalidated in the child. There is nothing we can (or should) do about it in the PKCS#11 engine. Actually, after attempting to forward authentication packets to the monitor which showed up as a dead end, we will use the /dev/crypto engine that has no fork issues. See the Evaluation notes for more information.