pk11_load_pubkey()/pk11_load_privkey() in OpenSSL PKCS#11 engine create a reference to OpenSSL RSA key (by calling pk11_get_public_rsa_key()/pk11_get_private_rsa_key()) but do not create a reference to object handle associated with the key. E.g. for pk11_load_pubkey():
1072 if (rsa)
1073 {
1074 check_new_rsa_key(sp, (void *) rsa);
1075
1076 h_pub_key = pk11_get_public_rsa_key(rsa, sp);
1077 if (h_pub_key == CK_INVALID_HANDLE)
1078 {
1079 EVP_PKEY_free(pkey);
1080 pkey = NULL;
1081 }
1082 }
...
1091 pk11_return_session(sp);
1092 return(pkey);
1093 }
The reference to the OpenSSL key is created like this (e.g. in pk11_get_public_rsa_key()):
1191 sp->rsa = rsa;
Thus, in pk11_load_pubkey()/pk11_load_privkey() the 'sp' structure is returned to the cache with sp->rsa set to the OpenSSL key but sp->rsa_{pub,priv}_key set to CK_INVALID_HANDLE (ensured by check_new_rsa_key()). Now if a thread comes along which happens to get the same cache entry and using the same RSA OpenSSL key (e.g. for RSA_verify()/RSA_sign()) a cache hit will result in the invalid handle to be used. The RSA operation will go boom.