High frequency of yield() system calls by large number of threads in parallel can stall timeshare class tick processing in the clock thread. The yield() system call acquires the thread dispatcher lock with thread_lock(), as does the clock thread. Contention for these
locks can slow the clock thread progress to a crawl and cause it to fall behind.
Commands that depend on callouts such as sleep hang, and when the threads calling yield() cease absurdly high load averages are reported as an artifact of the clock thread impact.
This behavior has been observed due to Java Parallel Garbage Collection.
Work Around
By default, one thread per cpu is allocated in the 1.5 JVM's for parallel garbage collection. The parameter -XX:ParallelGCThreads=N can be specified to limit the number of GC threads to a reasonable value on machines with a large number of cpu's. If the machine is running with multiple JVM's, this thread allocation should be considered across all JVM's.
Upgrade to JVM 1.5.0_10 or later. The JVM was modified in that
build to alternately yield() and sleep(), dramatically reducing
the occurrence of this bug.