OpenSolaris

A software partner switched their DNS server from using poll(2) to /dev/poll, and are experiencing intermittent ioctl(dpfd, DP_POLL,...) hangs much like described in bugid 6639835.   It also looks like bugid 6437867, where a related analysis of that bug states :
"What's failing here is that although udp_rcv_enqueue is calling pollwakeup on the received packet, the sleeping process just never wakes up.  It looks like some sort of timing problem in poll that could have been exposed by the new UDP design, but it's not clear to me."

Unfortunately, it looks like the application hanging was "patched" to poll for shorter intervals (pseudo busy loop) rather than fixing the ultimate bug triggering 6437867 as this person saw it, and some other related issues were fixed in bugid 6440104, so this bug was closed though likely the initial root cause persists.

Using some speculative DTrace scripting, I am able to show for this DNS server that, after dpioctl() enters dp_pcache_poll, upd_rcv_enqueue operations occur (verified by snoop as DNS traffic for this process), pollwakeup/pollhead_insert calls are made, but dpioctl() does not return until I unwedge the process via SIGSTOP/SIGCONT hundreds of milliseconds later.  The problem occurs quite rarely in general, but, adding credence to the belief it is a race condition, it tends to happen much more quickly when DTrace is run on the process.

This bug is reproducible even if all but one CPU core is enabled, and the DNS server is running as a single kernel thread.
That last line should state "even if all but one CPU core is *disabled*" (not enabled).  So the bug occurs even with only one CPU core online.

Could change the code calling ioctl(dpfd,DP_POLL,...) to loop with a shorter dp_timeout (perhaps 10 milliseconds) so if the bug hits we just have a small latency hiccup rather than a hang.

Or, if source is not available to recompile, could create a small library to interpose the ioctl() command and loop itself for DP_POLL.

Neither of these is a clean solution that will solve this for everyone, but might be okay to temporarily work around the problem in some cases.

Here is one possible implementation of a preloadable llibrary :
/* *************************************** */
/*                                                                              
 * Work around for intermittent /dev/poll ioctl(DP_POLL) hang....               
 * compile (SunStudio) :                                                        
 * cc -G -K pic [-xarch=amd64|-xarch=v9] -o ioctl[64].so -mt ioctl_LD_PRELOAD.c
 * Then set LD_PRELOAD_[32|64] environment variable to point to that .so before
 * running the binary that hangs with /dev/poll.                                
 */

#include <sys/types.h>
#include <sys/devpoll.h>
#include <stdio.h>
#include <stdarg.h>
#include <fcntl.h>
#include <dlfcn.h>
#include <stdlib.h>
#include <errno.h>
#include <thread.h>

/* set this to an acceptable latency in milliseconds */
/* could make an environment variable and put in an ini section */

static int oklatency=10; /* if bug hits we can be 10 msecs late */

int ioctl(int fd, int request, ...)
{
  va_list alist;
  static int (* fptr)(int, int, ...);
  static mutex_t myLock = DEFAULTMUTEX;
  struct dvpoll *dopoll;
  int timeout,timeout_left;
  int retval=0;

  va_start(alist, request);

  if (fptr == NULL) {
    mutex_lock(&myLock);
    if ((fptr = (int (*)(int,int,...))dlsym(RTLD_NEXT, "ioctl")) == NULL)  {
      fprintf(stderr,"dlsym: %s\n", dlerror()); exit (1);
    }
    mutex_unlock(&myLock);
    printf("initializing ioctl fptr=%x\n",fptr); /* might comment out */
  }

  if (request != DP_POLL) { /* just pass along */
    retval = fptr(fd,request, alist);
  } else {
    dopoll = va_arg(alist, struct dvpoll *);
    timeout=timeout_left=dopoll->dp_timeout;
    if (timeout_left != -1 && timeout_left <= oklatency) {
      /* again, just pass along */
      retval = fptr(fd,request, alist);
    } else {
      dopoll->dp_timeout = oklatency;
      while (((timeout == -1) || (timeout_left > 0)) && retval == 0) {
	/* Note : this isn't 100% accurate on timeout but simple/fast */
	if (timeout != -1) timeout_left -= oklatency;
	retval = fptr(fd,request, dopoll);
      }
      dopoll->dp_timeout = timeout; /* cannot leave with changed struct */
    }
  }

  va_end(alist);
  return retval;
}
/* *************************************** */

N/A